Stackrox/ACS reporting vulns in packages removed in later layers

[mtcolman]

Hi,

When scanning image node:18.16.1 with ACS it will give this violation:

      "violatedPolicies": [
        {
          "name": "Fixable Severity at least Important",
          "severity": "HIGH",
          "description": "Alert on deployments with fixable vulnerabilities with a Severity Rating at least Important",
          "violation": [
            "Fixable CVE-2022-25883 (CVSS 7.5) (severity Important) found in component 'semver' (version 7.3.8), resolved by version 7.5.2"
          ],
          "remediation": "Use your package manager to update to a fixed version in future builds or speak with your security team to mitigate the vulnerabilities.",
          "failingCheck": true
        },

The image has npm version 9.5.1:

$ podman run -it --entrypoint=/bin/bash node:18.16.1
root@37d1a308c700:/# npm --version
9.5.1

and it clearly uses semver 7.3.8:

root@37d1a308c700:/# npm -g list --depth=20 | grep semver
  | | `-- semver@7.3.8 deduped
  | | `-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | | +-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | | +-- semver@7.3.8 deduped
  | `-- semver@7.3.8 deduped
  | | | | `-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | `-- semver@7.3.8 deduped
  | +-- semver@7.3.8 deduped
  | `-- semver@7.3.8 deduped
  +-- semver@7.3.8
  |   `-- semver@7.3.8 deduped

I therefore upgrade npm to 9.8.0 to update semver to 7.5.2:

root@37d1a308c700:/# npm -g install npm@9.8.0

removed 17 packages, and changed 67 packages in 6s

28 packages are looking for funding
  run `npm fund` for details

root@37d1a308c700:/# npm --version
9.8.0

root@37d1a308c700:/# npm -g list --depth=20 | grep semver
  | | `-- semver@7.5.2 deduped
  | | `-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | | +-- semver@7.5.2 deduped
  | | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | `-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | `-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | `-- semver@7.5.2 deduped
  +-- semver@7.5.2
  |   `-- semver@7.5.2 deduped

And that aligns to where ACS says the fix is: "Fixable CVE-2022-25883 (CVSS 7.5) (severity Important) found in component 'semver' (version 7.3.8), resolved by version 7.5.2".

I create a new container with the npm upgrade within it and scan this with ACS. And the finding is still present:

If I check the package.json in the container, it shows it's version 7.5.2:

$ podman run namloc2001/nodesem:a cat /usr/local/lib/node_modules/npm/node_modules/semver/package.json | grep version
  "version": "7.5.2",
  "description": "The semantic version parser used by npm.",
    "version": "4.15.1",

And if I check globally the version of semver in the container, I get:

$ podman run namloc2001/nodesem:a npm list -g --depth=20 | grep semver
  | | `-- semver@7.5.2 deduped
  | | `-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | | +-- semver@7.5.2 deduped
  | | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | `-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | `-- semver@7.5.2 deduped
  | +-- semver@7.5.2 deduped
  | `-- semver@7.5.2 deduped
  +-- semver@7.5.2
  |   `-- semver@7.5.2 deduped

So it looks like ACS is finding something in a layer in the container, which is then resolved by a later layer

When I scan the updated image with trivy and grype, I don't get any semver alerts. And when I create an SBOM with syft, I get:

...
safe-buffer                         5.2.1                           npm
safer-buffer                        2.1.2                           npm
sed                                 4.9-1                           deb
semver                              7.5.2                           npm          <<<------------------- note it is showing at the correct 7.5.2 version
sensible-utils                      0.0.17+nmu1                     deb
set-blocking                        2.0.0                           npm
shared-mime-info                    2.2-1                           deb
shebang-command                     2.0.0                           npm
shebang-regex                       3.0.0                           npm
...

If I then scan again with Grype, using the -s AllLayers flag (otherwise default is "Squashed"), I get:

$ grype namloc2001/nodesem:a -s AllLayers
 ✔ Vulnerability DB        [updated]
 ✔ Parsed image
 ✔ Cataloged packages      [1282 packages]
 ✔ Scanning image...       [726 vulnerabilities]
   ├── 1 critical, 63 high, 144 medium, 30 low, 445 negligible (43 unknown)
   └── 34 fixed
NAME                          INSTALLED             FIXED-IN     TYPE  VULNERABILITY        SEVERITY
...
python3.11-minimal            3.11.2-6              (won't fix)  deb   CVE-2023-27043       Medium
semver                        7.3.8                 7.5.2        npm   GHSA-c2qf-rxjj-qqgw  Medium
tar                           1.34+dfsg-1.2                      deb   CVE-2005-2541        Negligible
...

I can't see a similar option for roxctl and I also don't now why roxctl wouldn't default to squashed too; the vulnerability isn't in my image if I've removed/upgraded the package. Here is the roxctl command I used:

roxctl image scan --force --endpoint https://localhost:8443 --image namloc2001/nodesem:a --insecure-skip-tls-verify

Here's the Dockerfile I've used:

FROM node:18.16.1
RUN npm install -g npm@9.8.0

[RTann]

Thank you for bringing this to our attention. This is very strange, as we do try to ensure we do not show results for packages which do not exist in the final image.

I have been able to reproduce this with the provided image, namloc2001/nodesem:a. However, I have also tried to reproduce this with my own image, which I created based on your instructions:

FROM node:18.16.1
RUN npm install -g npm@9.8.0

I have compared my image to yours, and I can only find one difference: mine is a Docker image (mediatype application/vnd.docker.distribution.manifest.v2+json) while yours is an OCI image (mediatype application/vnd.oci.image.config.v1+json). Because of this, the layer digests are different. The layer contents, should be the same, though. I even compared the Dockerfile instructions, and they both matched.

So, for some reason, your OCI image is not scanned correctly, while my Docker image is. We will need to investigate this further.

[RTann]

I believe I determined the cause of this:

my Docker image recreates the /usr/local/lib/node_modules/npm/mode_modules directory while your OCI image does not.

I inspected the layers, and I found the Docker image has /usr/local/lib/node_modules/npm/.wh..wh..opq in the top layer while the OCI image has a ton of /.wh.<whatever> files. So, it seems like Docker opts to recreate the entire /usr/local/lib/node_modules/npm/mode_modules while OCI opts to update it (remove files which are no longer needed in the upper layers and just edit existing files).

So, StackRox Scanner correctly does not say semver has any issues in my image, as it knows to ignore the older version in the lower layer. StackRox Scanner does not ignore it in the OCI image, though, because of a bug. We only add "language-level" packages to languageFeatureMap if they are vulnerable. semver 7.5.2 is not vulnerable to anything, so it is not added to the map. Eventually, we see that we have semver 7.3.8 in a lower layer which is vulnerable to something, so we check the map to find it does not have an entry for the same file path, so we add it.

Thank you for bringing this to our attention. We will track the progress of this internally.

[mtcolman]

@RTann is there any update on this please?

[RTann]

@mtcolman we hae shifted our efforts to working on a new Scanner, based on ClairCore, which will be released sometime soon in the future. That being said, it looks like the new scanner has this same issue. See https://issues.redhat.com/browse/CLAIRDEV-6