retry-on-snapshot-warnings - not working as expected on separate snapshot/review workflows #632

felickz · 2023-12-04T18:35:14Z

Using retry-on-snapshot-warnings for a submission from a different workflow as described in the docs. If the snapshot upload completes during the phase where the review task is waiting for an upload against the head SHA - none of the retries pick it up. If you re-run the review workflow it picks up the newly committed snapshot.

On Push:

Submission Workflow - Submits dependency snapshot (takes 2m 3s to complete)

On PR:

Review Workflow - run 1 - Dependency review with retry and long timeout (Retry timeout exceeded. Proceeding... after 4m 37s)

Submission Workflow

after 2m submits its dependency snapshot for the commit 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f
logs_22.zip

Snapshot submission to "sha": "63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f" + "ref": "refs/heads/feature/FSharp-Data"

2023-12-04T18:10:54.1830002Z ##[notice]Submitting snapshot...
2023-12-04T18:10:54.1858756Z ##[notice]{
    "detector": {
        "name": "Component Detection",
        "version": "0.0.1",
        "url": "https://github.com/advanced-security/component-detection-dependency-submission-action"
    },
    "version": 0,
    "job": {
        "correlator": "dependency-submission",
        "id": "7090712486"
    },
    "sha": "63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f",
    "ref": "refs/heads/feature/FSharp-Data",

... and completes within a second at 2023-12-04T18:10:55.414Z:

2023-12-04T18:10:55.4988607Z ##[notice]Snapshot successfully created at 2023-12-04T18:10:55.414Z

Review Workflow

logs_23.zip

ref: refs/pull/8/merge
base ref: master
head ref: feature/FSharp-Data

2023-12-04T18:09:23.5277218Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:23.5277924Z Retrying in 10 seconds...
2023-12-04T18:09:33.8899944Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:33.8900873Z Retrying in 10 seconds...
2023-12-04T18:09:44.4259065Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:44.4259945Z Retrying in 10 seconds...
2023-12-04T18:09:54.7741812Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:54.7742670Z Retrying in 10 seconds...
2023-12-04T18:10:05.1531584Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:05.1533086Z Retrying in 10 seconds...
2023-12-04T18:10:15.4874900Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:15.4876137Z Retrying in 10 seconds...
2023-12-04T18:10:25.8345344Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:25.8346111Z Retrying in 10 seconds...
2023-12-04T18:10:36.1931593Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:36.1932274Z Retrying in 10 seconds...
2023-12-04T18:10:46.5429500Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:46.5430353Z Retrying in 10 seconds...

at this point the snapshot has been submitted
- for the next 2m30s looking for snapshot but not finding it

2023-12-04T18:10:56.9246186Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:56.9246809Z Retrying in 10 seconds...
2023-12-04T18:11:07.2595829Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:07.2596661Z Retrying in 10 seconds...
2023-12-04T18:11:17.6323447Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:17.6324235Z Retrying in 10 seconds...
2023-12-04T18:11:27.9848187Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:27.9849254Z Retrying in 10 seconds...
2023-12-04T18:11:38.3612135Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:38.3612905Z Retrying in 10 seconds...
2023-12-04T18:11:48.7582440Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:48.7583606Z Retrying in 10 seconds...
2023-12-04T18:11:59.1062944Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:59.1063681Z Retrying in 10 seconds...
2023-12-04T18:12:09.4695845Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:09.4696654Z Retrying in 10 seconds...
2023-12-04T18:12:19.8232333Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:19.8232973Z Retrying in 10 seconds...
2023-12-04T18:12:30.1726324Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:30.1727051Z Retrying in 10 seconds...
2023-12-04T18:12:40.4742149Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:40.4744475Z Retrying in 10 seconds...
2023-12-04T18:12:50.7921268Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:50.7921899Z Retrying in 10 seconds...
2023-12-04T18:13:01.1266767Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:01.1267423Z Retrying in 10 seconds...
2023-12-04T18:13:11.4858149Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:11.4858916Z Retrying in 10 seconds...
2023-12-04T18:13:21.8385364Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:21.8386034Z Retrying in 10 seconds...
2023-12-04T18:13:32.1692568Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:32.1693442Z Retry timeout exceeded. Proceeding...
2023-12-04T18:13:32.6080649Z Dependency review did not detect any denied packages

Review Workflow - run 2

Most interesting is that re-running dependency review task at any point in the future succeeds after 2 tries (doesnt mention a snapshot found but looking at the detections it has found dependencies that only exist in the snapshot manifest):

Review Workflow - run2
- logs_24.zip

ref: refs/pull/8/merge
base ref: master
head ref: feature/FSharp-Data

2023-12-04T18:30:08.1275306Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:30:08.1276356Z Retrying in 10 seconds...
2023-12-04T18:30:18.3518175Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:30:18.3518906Z Retrying in 10 seconds...
2023-12-04T18:30:30.0878871Z ##[debug]Filtered Changes: [{"change_type":"added","manifest":"Fsharp-WebAPI.fsproj","ecosystem":"nuget","name":"FSharp.Data","version":"6.3.0","package_url":"pkg:nuget/FSharp.Data@6.3.0","license":null,"source_repository_url":"https://github.com/fsprojects/FSharp.Data","scope":"runtime","vulnerabilities":[]},{"change_type":"added","manifest":"/Fsharp-WebAPI.fsproj","ecosystem":"nuget","name":"FSharp.Data","version":"6.3.0","package_url":"pkg:nuget/FSharp.Data@6.3.0","license":null,"source_repository_url":"https://github.com/fsprojects/FSharp.Data","scope":"runtime","vulnerabilities":[]},{"change_type":"added","manifest":"/Fsharp-

The text was updated successfully, but these errors were encountered:

juxtin · 2023-12-06T17:09:05Z

Hi @felickz, I really appreciate the reproduction and detailed logs! Much of this does sound very strange to me, but one thing sticks out as especially strange:

Most interesting is that re-running dependency review task at any point in the future succeeds after 2 tries

Can you clarify that this applies truly at any point? In other words, if you wait for a very long time (like an hour), does it still retry at least once before succeeding?

felickz · 2023-12-06T21:25:56Z

Hi @felickz, I really appreciate the reproduction and detailed logs! Much of this does sound very strange to me, but one thing sticks out as especially strange:

Most interesting is that re-running dependency review task at any point in the future succeeds after 2 tries

Can you clarify that this applies truly at any point? In other words, if you wait for a very long time (like an hour), does it still retry at least once before succeeding?

In the sample is showed - it was @ 15 mins:

2023-12-04T18:13:32.1693442Z Retry timeout exceeded. Proceeding...
Re-run
- 2023-12-04T18:30:30.0878871Z - retry succeed

In a previous repro - i had run another time with 5 minutes between with similar behavior, but rerun did not mention any no snapshots found - (initial run):

2023-12-04T17:59:17.1762701Z Retry timeout exceeded. Proceeding...
Re-Run
- 2023-12-04T18:04:56.0137937Z Dependency review did not detect any denied packages

juxtin · 2023-12-06T22:48:53Z

@febuiles I don't have a ton of time to devote to this and nothing's really jumping out at me, but I do have a vague suspicion that this may be related to https://github.com/github/dependency-snapshots-api/pull/615. I thought that would only change the way we handle new canonical snapshots and that shouldn't matter here, but if there's some reason why it's taking a while for new snapshots to be written to the ds_snapshots table then that could explain what's going on here.

febuiles · 2023-12-07T06:24:50Z

@juxtin thanks for the extra feedback 🙇

@jamisonhyatt any ideas if the locks-related change could be having an impact downstream like this? Does any of this look suspicious to you?

felickz · 2024-03-15T04:20:27Z

I have another oddity where it seems submission actions like anchore/sbom-action are using a merge commit when they upload a snapshot when running as an on pull_request trigger:

  ##[debug]  "eventName": "pull_request",
  ##[debug]  "sha": "6f50a7568be909f93a78e164a90362a365f92a44",
  ##[debug]  "ref": "refs/pull/22/merge",
  ##[debug]  "workflow": "Syft SBOM Action",
  ##[debug]  "action": "__anchore_sbom-action_2",
  ##[debug]  "actor": "felickz",
  ##[debug]  "job": "buildAndUpload",
  ##[debug]  "runNumber": 214,
  ##[debug]  "runId": 8290771455,

Then the DR action (with on pull_request trigger) is looking for sha of the latest commit in the HEAD branch.
Leading to: No snapshots were found for the head SHA a9e00023489e612bb5cfbc81ea97202a30af124e.

Is this something the toolkit is intended to handle?

I am not totally clear what this sha is showing me, this might be a temporary hidden merge branch that is used to check "This branch has no conflicts with the base branch"? Would it make any sense for DR to be also looking at this SHA for snapshot submissions?

jovel · 2024-03-15T16:20:57Z

Need to do research if the issue is with the backend service or potential race condition

rognan · 2024-03-21T01:25:45Z

I also encountered this problem, only I'm using the gradle/actions/dependency-submission action to submit the dependency snapshot.

With an added known vulnerable dependency I could re-run the review workflow at least 3 times all ending in a timeout and no vulnerabilites reported. The timeout was set to 10 minutes and the three times were run consecutively ...

Then, after waiting for maybe 30+ minutes (searching for answers and coming back for debug-logs), the re-runs suddenly started consistently reporting a "vulnerable dependency detected". It reminded me of how replication lag behaves as the snapshot propagates across data stores ..

I couldn't find an option for failing the build on timeouts, and with the green checkmark on the PR I doubt many people will study the logs to see if it all ran as they expected.

Should it fail, or at least give a warning, when the timeout expires and there's still no snapshot?

Pil0tXia · 2024-04-24T07:22:37Z

Hello everyone,

I believe this issue with the dependency-review-action that renders it completely unusable for Gradle projects.

Following the documentation for gradle-build-action, setup-gradle, and dependency-submission, I've configured the following workflow files:

Generating a dependency graph and uploading it to artifacts: code-scanning.yml; Run log
Waiting for the first step to complete, then downloading the dependency graph from artifacts and submitting it: dependency-graph.yml; Run log
Waiting for the second step to complete, updating snapshots, and checking if all introduced/updated dependencies in the PR are Apache-2.0 compatible: license.yml; Run log

However, even though the second step has been completed, the dependency-review-action consistently outputs the following log:

Retrying in 10 seconds...
No snapshots were found for the head SHA 96f39e13dc6d1db148fc6e8cacaac18fdb2ae285.
Retry timeout exceeded. Proceeding...
Dependency review did not detect any denied packages

Additionally, the dependency-review-action fails to detect any changes in Java dependencies in this PR.

I kindly request the community to address this issue as soon as possible. My PR, which I've invested considerable research time into, is currently stalled because of this issue. Thank you very much.

Pil0tXia · 2024-04-24T08:37:38Z

I tried to lengthen retry-on-snapshot-warnings-timeout to 3600 (1hr) just now and it seems dependency-review-action got snapshot change after 25m 49s in this run.

However, the time taken by the dependency-review-action to detect snapshot changes is not fixed. e.g. in this run, it took 50 minutes.

jonjanego added the bug Something isn't working label Mar 15, 2024

jovel added the needs research label Mar 15, 2024

Pil0tXia mentioned this issue Apr 24, 2024

[ISSUE #4720] Modernize CI license check and Enable Dependabot apache/eventmesh#4827

Open

Pil0tXia mentioned this issue May 8, 2024

[Question] Inconsistent gradle configuration exclusion results gradle/actions#198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

retry-on-snapshot-warnings - not working as expected on separate snapshot/review workflows #632

retry-on-snapshot-warnings - not working as expected on separate snapshot/review workflows #632

felickz commented Dec 4, 2023 •

edited

juxtin commented Dec 6, 2023

felickz commented Dec 6, 2023 •

edited

juxtin commented Dec 6, 2023

febuiles commented Dec 7, 2023

felickz commented Mar 15, 2024 •

edited

jovel commented Mar 15, 2024 •

edited

rognan commented Mar 21, 2024

Pil0tXia commented Apr 24, 2024

Pil0tXia commented Apr 24, 2024 •

edited

retry-on-snapshot-warnings - not working as expected on separate snapshot/review workflows #632

retry-on-snapshot-warnings - not working as expected on separate snapshot/review workflows #632

Comments

felickz commented Dec 4, 2023 • edited

Submission Workflow

Review Workflow

Review Workflow - run 2

juxtin commented Dec 6, 2023

felickz commented Dec 6, 2023 • edited

juxtin commented Dec 6, 2023

febuiles commented Dec 7, 2023

felickz commented Mar 15, 2024 • edited

jovel commented Mar 15, 2024 • edited

rognan commented Mar 21, 2024

Pil0tXia commented Apr 24, 2024

Pil0tXia commented Apr 24, 2024 • edited

felickz commented Dec 4, 2023 •

edited

felickz commented Dec 6, 2023 •

edited

felickz commented Mar 15, 2024 •

edited

jovel commented Mar 15, 2024 •

edited

Pil0tXia commented Apr 24, 2024 •

edited