Can't use retryStrategy AND hooks on a step that outputs an artifact with downstream consumers #12109
Closed
2 of 3 tasks
Assignees
Labels
area/hooks
area/retryStrategy
Template-level retryStrategy
area/templating
Templating with `{{...}}`
P2
Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important
type/bug
Milestone
Comments
|
I should explain that the python script is trying to randomly take too long and fail with the timeout about 50% of the time. I inadvertently left the commented out system.fail() in there, sorry. |
|
I think it might be related to this other issue that was recently fixed, not sure. |
|
It also fails to resolve the artifact with a regular exit handler (hook named exit) only. |
agilgur5
added
area/hooks
area/retry-manual
3 tasks
agilgur5
added
P2
|
I was able to reproduce this issue using the latest version of Argo Workflows (v3.5.4), which includes #12192. Posting a few examples for clarity on the issue that we're experiencing. Outputs are properly recognized for steps that use EITHER lifecycle hooks, or retryStrategy: metadata:
generateName: artifact-hooks-
spec:
entrypoint: all-steps
templates:
- name: all-steps
steps:
- - name: step-one
inline:
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "touch /output.txt"
outputs:
artifacts:
- name: output-file
path: /output.txt
hooks:
success:
expression: 'steps["step-one"].status == "Succeeded"'
template: exit-zero
- - name: step-two
inline:
inputs:
artifacts:
- name: input-file
path: /input.txt
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "exit 0"
arguments:
artifacts:
- name: input-file
from: "{{steps.step-one.outputs.artifacts.output-file}}"
- name: exit-zero
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "exit 0"
metadata:
generateName: artifact-retries-
spec:
entrypoint: all-steps
templates:
- name: all-steps
steps:
- - name: step-one
inline:
retryStrategy:
limit: "3"
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "touch /output.txt"
outputs:
artifacts:
- name: output-file
path: /output.txt
- - name: step-two
arguments:
artifacts:
- name: input-file
from: "{{steps.step-one.outputs.artifacts.output-file}}"
inline:
inputs:
artifacts:
- name: input-file
path: /input.txt
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "exit 0"
But if the step with outputs uses retries AND hooks, then the step after it does not recognize the output of the first step: metadata:
generateName: artifact-retries-hooks-
spec:
entrypoint: all-steps
templates:
- name: all-steps
steps:
- - name: step-one
inline:
retryStrategy:
limit: "3"
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "touch /output.txt"
outputs:
artifacts:
- name: output-file
path: /output.txt
hooks:
success:
expression: 'steps["step-one"].status == "Succeeded"'
template: exit-zero
- - name: step-two
inline:
inputs:
artifacts:
- name: input-file
path: /input.txt
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "exit 0"
arguments:
artifacts:
- name: input-file
from: "{{steps.step-one.outputs.artifacts.output-file}}"
- name: exit-zero
container:
name: main
image: argoproj/argosay:v2
command:
- /bin/bash
- "-c"
args:
- "exit 0"
The failed node includes the following message: |
shuangkun
added a commit
to shuangkun/argo-workflows
that referenced
this issue
Mar 18, 2024
…2109 Signed-off-by: shuangkun <tsk2013uestc@163.com>
|
It should be that the hook node was selected probabilistically when building LocalScope. |
shuangkun
added a commit
to shuangkun/argo-workflows
that referenced
this issue
Mar 18, 2024
…2109 Signed-off-by: shuangkun <tsk2013uestc@163.com>
shuangkun
added a commit
to shuangkun/argo-workflows
that referenced
this issue
Mar 19, 2024
…2109 Signed-off-by: shuangkun <tsk2013uestc@163.com>
juliev0
pushed a commit
that referenced
this issue
Mar 30, 2024
agilgur5
added
area/retryStrategy
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Pre-requisites
:latestWhat happened/what you expected to happen?
I would expect for me to be able to use both hooks and retries on the same step. When I use one or the other, it works. But together, it messes up the resolution of artifacts for my downstream consumers. The error shows on the workflow as
unable to resolve references: Unable to resolve: "steps.build.outputs.artifacts.result". Additionally, the build step shows as Running but all the pods have completed or errored.Version
v3.5.0
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container
The text was updated successfully, but these errors were encountered: