fix containerized function mounts issue #4489

natasha41575 · 2022-02-25T02:27:35Z

Kustomize was not mounting files into containerized functions directly because the StorageMounts field was not being properly propagated to the function filter.

This PR allows users to use the StorageMounts fields, with the following requirements:

mount paths cannot be absolute paths
mount paths must be relative to the kustomization that refers to them
mount paths must be under the kustomization directory that refers to them

As a side note, if we are going to be promoting functions in kustomize we should probably build up a strong test suite for containerized functions and run those tests in CI. All tests needing docker are currently being skipped.

/cc @yuwenma
/cc @KnVerey

k8s-ci-robot · 2022-02-25T02:28:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: natasha41575

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [natasha41575]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

api/go.mod

api/krusty/fnplugin_test.go

api/krusty/testdata/charts/helloworld-chart/Chart.yaml

api/krusty/testdata/charts/helloworld-chart/values.yaml

yuwenma · 2022-02-25T02:36:27Z

Some minor comments. Looks great!

k8s-ci-robot · 2022-02-25T18:53:29Z

@natasha41575: This PR has multiple commits, and the default merge method is: merge.
You can request commits to be squashed using the label: tide/merge-method-squash

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

api/krusty/fnplugin_test.go

KnVerey · 2022-02-25T20:09:40Z

I poked around the history of the problem file, and it seems to me this never worked. Is that correct?

If so, if we were to release this PR, it would introduce a new security issue that (intended or not) we didn't have before.
Since container functions ARE enabled in kubectl, we should be very careful about this decision.

It looks like the feature was originally intended to have restrictions on the path, but that did not get implemented: #1902

volumes are relative to the Resource config
volumes must be in a directory under the config directory (no ../)

The way Plugin Graduation proposes to handle this is through Catalog, where the Catalog entry must specify allowed mounts. Inline config like what we're turning on here actually wouldn't be supported at all anymore:

If a containerized plugin needs network or disk access, its catalog entry MUST specify it. Redundant inline provider config cannot be used to grant it. When the catalog does specify it, no additional flags or config fields will be required to grant it the required access.

natasha41575 · 2022-03-18T18:24:10Z

I poked around the history of the problem file, and it seems to me this never worked. Is that correct?

It seems that this is the case, and I agree that you're right, it probably isn't great to allow an unrestricted path to be mounted in the container.

However, this lack of mounts would block usage of the render-helm-chart KRM function for local charts and I don't think it would be practical to wait for the Plugin Graduation and Catalog to be completed before asking users to migrate from the builtin Helm plugin.

WDYT about restricting mounted paths to the same load restrictions as all other paths (i.e. it must be in the current kustomization directory)? If we do that, would that alleviate your concerns regarding security?

KnVerey · 2022-03-22T22:34:30Z

WDYT about restricting mounted paths to the same load restrictions as all other paths (i.e. it must be in the current kustomization directory)? If we do that, would that alleviate your concerns regarding security?

Yes, this would work for me.

natasha41575 · 2022-04-04T00:50:50Z

@KnVerey awesome, PR is updated, with a check to ensure that the mounted paths are in the current directory. I added a test TestFnContainerMountsLoadRestrictions for this restriction as well.

KnVerey · 2022-04-05T15:53:24Z

As a side note, if we are going to be promoting functions in kustomize we should probably build up a strong test suite for containerized functions and run those tests in CI. All tests needing docker are currently being skipped.

As part of #4564 I noticed that it seems like some Docker-based tests do run in CI. But only the ones in cmd/config, only if they're gated with KUSTOMIZE_DOCKER_E2E specifically, and only on Linux: https://github.com/kubernetes-sigs/kustomize/blob/master/.github/workflows/go.yml#L61. Since we aren't running in verbose mode right now you can't see it in this PR's tests, but check out this test run from my PR: https://github.com/kubernetes-sigs/kustomize/runs/5826226816?check_suite_focus=true#step:4:846

natasha41575 · 2022-04-11T22:47:05Z

api/internal/plugins/loader/loader.go

@@ -49,7 +49,9 @@ func (l *Loader) Config() *types.PluginConfig {

 // SetWorkDir sets the working directory for this loader's plugins
 func (l *Loader) SetWorkDir(wd string) {
-	l.pc.FnpLoadingOptions.WorkingDir = wd
+	if wd != string(filepath.Separator) {
+		l.pc.FnpLoadingOptions.WorkingDir = wd


Without this change, TestFnContainerEnvVars fails with the error couldn't execute function: root working directory '/' not allowed. ldr.Root() returns "/" when run at the root-level, i.e. whenever we do th.Run(".") in our tests. If we don't set the working dir here, then the function filter uses the current working directory, which I believe is correct behavior in the cases that hit this problem.

Alternatively I think we could rewrite TestFnContainerEnvVars so that it does th.Run("dir") or something, and then we won't have to have this change here.

Hmm, but isn't this just bypassing a real restriction we have in our function code? In other words, won't it really enable the root directory to be used as valid plugin load location, effectively allowing plugins from anywhere on the disk?

If we don't set the working dir here, then the function filter uses the current working directory, which I believe is correct behavior in the cases that hit this problem.

Can you explain more why you believe this is correct? My inclination would be that it's easier to understand what's happening if the behavior is always the same (kustomization root, not invocation dir).

Alternatively I think we could rewrite TestFnContainerEnvVars so that it does th.Run("dir") or something, and then we won't have to have this change here.

Is it true that the tests in question are always using a virtual filesystem, and that's why it's not bad for the Kustomization dir to be the root? If so, that change will make the tests more realistic, right?

🤦‍♀️ you're right on all points. I think my brain was in another location when I wrote this part, for some reason I was thinking of the working directory as the kustomization root, which is obviously not the case and is the whole reason we are doing this anyways.

I will update the test.

Test updated! I had to change it to use a temporary directory in the real file system because it's shelling out to docker and needs the working directory to be a real one.

kyaml/fn/runtime/container/container_test.go

api/krusty/testdata/fnmounts/kustomization.yaml

natasha41575 · 2022-04-12T18:14:08Z

According to the logs, the docker tests are running in the linux job but not the mac one. Filed #4579

KnVerey · 2022-04-14T22:06:15Z

api/krusty/testdata/fnmounts/charts/helloworld-chart/values.yaml

@@ -0,0 +1,27 @@
+replicaCount: 1


Why two values files? And do we actually need the whole chart for the purposes of the test?

Pared this down to a minimal example!

kyaml/fn/runtime/container/container_test.go

KnVerey · 2022-04-14T22:48:28Z

api/krusty/fnplugin_test.go

+	assert.NoError(t, err)
+
+	testDir := filepath.Join(path, "testdata", "fnmounts")
+	command := exec.Command("kustomize", "build", testDir, "--enable-alpha-plugins")


Why is this test exec'ing Kustomize instead of using the test framework? How do we know the binary that will be found corresponds to the version of the code we think we're testing?

Originally I was trying to copy the entire testdata/fnmounts directory into the temp directory where the test framework was running, which turned out to be a headache.

But with a pared down helm chart (per your other suggestion), this is much easier to do with the test framework. Updated!

api/krusty/fnplugin_test.go

kyaml/runfn/runfn.go

KnVerey · 2022-04-18T21:17:47Z

/lgtm

k8s-ci-robot requested review from KnVerey and yuwenma February 25, 2022 02:27

k8s-ci-robot added cncf-cla: yes size/XL labels Feb 25, 2022

k8s-ci-robot added the approved label Feb 25, 2022

yuwenma reviewed Feb 25, 2022

View reviewed changes

api/go.mod Outdated Show resolved Hide resolved

natasha41575 mentioned this pull request Feb 25, 2022

Container KRM Mounts are not mounting via function parameters #4290

Closed

yuwenma reviewed Feb 25, 2022

View reviewed changes

api/krusty/fnplugin_test.go Outdated Show resolved Hide resolved

yuwenma reviewed Feb 25, 2022

View reviewed changes

api/krusty/testdata/charts/helloworld-chart/Chart.yaml Outdated Show resolved Hide resolved

yuwenma reviewed Feb 25, 2022

View reviewed changes

api/krusty/testdata/charts/helloworld-chart/values.yaml Outdated Show resolved Hide resolved

natasha41575 added the tide/merge-method-squash label Feb 25, 2022

natasha41575 requested a review from yuwenma February 25, 2022 18:53

k8s-ci-robot added size/L and removed size/XL labels Feb 25, 2022

KnVerey reviewed Feb 25, 2022

View reviewed changes

api/krusty/fnplugin_test.go Outdated Show resolved Hide resolved

natasha41575 requested a review from KnVerey March 18, 2022 18:24

natasha41575 force-pushed the FnMountIssue branch from 3807213 to 943e3d9 Compare April 4, 2022 00:47

k8s-ci-robot added size/XL and removed size/L labels Apr 4, 2022

natasha41575 force-pushed the FnMountIssue branch from 943e3d9 to 35d6f33 Compare April 4, 2022 00:48

natasha41575 force-pushed the FnMountIssue branch 2 times, most recently from 5f65d69 to 98ccc5e Compare April 4, 2022 01:23

natasha41575 force-pushed the FnMountIssue branch 2 times, most recently from 56a5aab to 6f56f95 Compare April 11, 2022 22:44

natasha41575 commented Apr 11, 2022

View reviewed changes

fix containerized function mounts issue

6ccb493

natasha41575 force-pushed the FnMountIssue branch from 6f56f95 to 6ccb493 Compare April 11, 2022 22:55

skip path test on windows

4fdbebf

natasha41575 commented Apr 11, 2022

View reviewed changes

kyaml/fn/runtime/container/container_test.go Outdated Show resolved Hide resolved

natasha41575 force-pushed the FnMountIssue branch 2 times, most recently from 473d005 to 864b0e0 Compare April 11, 2022 23:36

natasha41575 commented Apr 11, 2022

View reviewed changes

api/krusty/testdata/fnmounts/kustomization.yaml Outdated Show resolved Hide resolved

natasha41575 force-pushed the FnMountIssue branch from 864b0e0 to 94c2cca Compare April 11, 2022 23:42

move test out of temp dir

96977a1

natasha41575 force-pushed the FnMountIssue branch from 94c2cca to 96977a1 Compare April 11, 2022 23:51

natasha41575 requested a review from KnVerey April 11, 2022 23:51

update tests to deal with new working dir restrictions

dfaa43c

natasha41575 force-pushed the FnMountIssue branch from 28f5cce to dfaa43c Compare April 14, 2022 16:55

KnVerey reviewed Apr 14, 2022

View reviewed changes

k8s-ci-robot added size/L and removed size/XL labels Apr 15, 2022

natasha41575 force-pushed the FnMountIssue branch 2 times, most recently from 5ac4df0 to 008ca9a Compare April 15, 2022 17:19

code review

e0166b5

natasha41575 force-pushed the FnMountIssue branch from 008ca9a to e0166b5 Compare April 15, 2022 17:24

natasha41575 requested a review from KnVerey April 15, 2022 17:34

k8s-ci-robot assigned KnVerey Apr 18, 2022

k8s-ci-robot added the lgtm label Apr 18, 2022

k8s-ci-robot merged commit 9d5491c into kubernetes-sigs:master Apr 18, 2022

natasha41575 deleted the FnMountIssue branch April 18, 2022 22:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix containerized function mounts issue #4489

fix containerized function mounts issue #4489

natasha41575 commented Feb 25, 2022 •

edited

Loading

k8s-ci-robot commented Feb 25, 2022

yuwenma commented Feb 25, 2022

k8s-ci-robot commented Feb 25, 2022

KnVerey commented Feb 25, 2022 •

edited

Loading

natasha41575 commented Mar 18, 2022

KnVerey commented Mar 22, 2022

natasha41575 commented Apr 4, 2022

KnVerey commented Apr 5, 2022

natasha41575 Apr 11, 2022 •

edited

Loading

KnVerey Apr 14, 2022

natasha41575 Apr 14, 2022

natasha41575 Apr 14, 2022

natasha41575 commented Apr 12, 2022

KnVerey Apr 14, 2022

natasha41575 Apr 15, 2022

KnVerey Apr 14, 2022

natasha41575 Apr 15, 2022

KnVerey commented Apr 18, 2022

fix containerized function mounts issue #4489

fix containerized function mounts issue #4489

Conversation

natasha41575 commented Feb 25, 2022 • edited Loading

k8s-ci-robot commented Feb 25, 2022

yuwenma commented Feb 25, 2022

k8s-ci-robot commented Feb 25, 2022

KnVerey commented Feb 25, 2022 • edited Loading

natasha41575 commented Mar 18, 2022

KnVerey commented Mar 22, 2022

natasha41575 commented Apr 4, 2022

KnVerey commented Apr 5, 2022

natasha41575 Apr 11, 2022 • edited Loading

Choose a reason for hiding this comment

KnVerey Apr 14, 2022

Choose a reason for hiding this comment

natasha41575 Apr 14, 2022

Choose a reason for hiding this comment

natasha41575 Apr 14, 2022

Choose a reason for hiding this comment

natasha41575 commented Apr 12, 2022

KnVerey Apr 14, 2022

Choose a reason for hiding this comment

natasha41575 Apr 15, 2022

Choose a reason for hiding this comment

KnVerey Apr 14, 2022

Choose a reason for hiding this comment

natasha41575 Apr 15, 2022

Choose a reason for hiding this comment

KnVerey commented Apr 18, 2022

natasha41575 commented Feb 25, 2022 •

edited

Loading

KnVerey commented Feb 25, 2022 •

edited

Loading

natasha41575 Apr 11, 2022 •

edited

Loading