DAG with continueOn in error after retry

[fstaudt]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I created following DAG with some tasks that should be bypassed if they fail.
I used continueOn keyword on DAG task for this purpose.
I also tried to use depends keyword on DAG tasks and got the same result.

First execution of workflow looks like this (it is the expected result):

When I retry this workflow, the DAG workflow is in Error with following message:
Ancestor task node continue not found

Workflow after retry looks like this (not expected):

What I expect after retry is that:

• failed tasks with dependent tasks already executed remain failed in DAG
• failed tasks without dependent tasks already executed are retried

Version

v3.4.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: "dag-to-retry"
spec:
  podMetadata:
    annotations:
      sidecar.istio.io/inject: "false"
  entrypoint: dag
  templates:
    - name: step
      inputs:
        parameters:
          - name: exitCode
      container:
        image: alpine:3.7
        command: [ sh, "-c", "exit {{inputs.parameters.exitCode}}" ]
    - name: dag
      dag:
        failFast: false
        tasks:
          - name: success
            template: step
            arguments:
              parameters:
                - name: exitCode
                  value: 0
          - name: failure
            template: step
            dependencies:
              - success
            arguments:
              parameters:
                - name: exitCode
                  value: 1
          - name: task-after-failure
            template: step
            dependencies:
              - failure
            arguments:
              parameters:
                - name: exitCode
                  value: 0
          - name: continue
            template: step
            continueOn:
              failed: true
            dependencies:
              - success
            arguments:
              parameters:
                - name: exitCode
                  value: 2
          - name: task-after-continue
            template: step
            dependencies:
              - continue
            arguments:
              parameters:
                - name: exitCode
                  value: 0

Logs from the workflow controller

$ kubectl logs -n argo deploy/argo-workflows-workflow-controller | grep dag-to-retry-6jqss
Found 2 pods, using pod/argo-workflows-workflow-controller-65f48b9fdb-zrb6l
time="2023-07-19T16:30:16.101Z" level=info msg="Processing workflow" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.119Z" level=info msg="Updated phase  -> Running" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.120Z" level=info msg="DAG node dag-to-retry-6jqss initialized Running" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.120Z" level=info msg="All of node dag-to-retry-6jqss.success dependencies [] completed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.120Z" level=info msg="Pod node dag-to-retry-6jqss-1212128117 initialized Pending" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.165Z" level=info msg="Created pod: dag-to-retry-6jqss.success (dag-to-retry-6jqss-step-1212128117)" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.166Z" level=info msg="TaskSet Reconciliation" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.166Z" level=info msg=reconcileAgentPod namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:16.173Z" level=info msg="Workflow update successful" namespace=core-support phase=Running resourceVersion=43610951 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.101Z" level=info msg="Processing workflow" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.101Z" level=info msg="Task-result reconciliation" namespace=core-support numObjs=0 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.101Z" level=info msg="node changed" namespace=core-support new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-to-retry-6jqss-1212128117 old.message= old.phase=Pending old.progress=0/1 workflow
=dag-to-retry-6jqss
time="2023-07-19T16:30:26.102Z" level=info msg="All of node dag-to-retry-6jqss.continue dependencies [success] completed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.103Z" level=info msg="Pod node dag-to-retry-6jqss-3259869403 initialized Pending" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.455Z" level=info msg="Created pod: dag-to-retry-6jqss.continue (dag-to-retry-6jqss-step-3259869403)" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.455Z" level=info msg="All of node dag-to-retry-6jqss.failure dependencies [success] completed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.455Z" level=info msg="Pod node dag-to-retry-6jqss-3193705776 initialized Pending" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.656Z" level=info msg="Created pod: dag-to-retry-6jqss.failure (dag-to-retry-6jqss-step-3193705776)" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.656Z" level=info msg="TaskSet Reconciliation" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.656Z" level=info msg=reconcileAgentPod namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:26.670Z" level=info msg="Workflow update successful" namespace=core-support phase=Running resourceVersion=43611257 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.456Z" level=info msg="Processing workflow" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="Task-result reconciliation" namespace=core-support numObjs=0 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="Pod failed: Error (exit code 1)" displayName=failure namespace=core-support pod=dag-to-retry-6jqss-step-3193705776 templateName=step workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="node changed" namespace=core-support new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=dag-to-retry-6jqss-3193705776 old.message= old.phase=Pending old.pro
gress=0/1 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="node unchanged" namespace=core-support nodeID=dag-to-retry-6jqss-1212128117 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="Pod failed: Error (exit code 2)" displayName=continue namespace=core-support pod=dag-to-retry-6jqss-step-3259869403 templateName=step workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="node changed" namespace=core-support new.message="Error (exit code 2)" new.phase=Failed new.progress=0/1 nodeID=dag-to-retry-6jqss-3259869403 old.message= old.phase=Pending old.pro
gress=0/1 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.457Z" level=info msg="All of node dag-to-retry-6jqss.task-after-continue dependencies [continue] completed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.459Z" level=info msg="Pod node dag-to-retry-6jqss-2496561594 initialized Pending" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.473Z" level=info msg="Created pod: dag-to-retry-6jqss.task-after-continue (dag-to-retry-6jqss-step-2496561594)" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.473Z" level=info msg="Skipped node dag-to-retry-6jqss-3806205299 initialized Omitted (message: omitted: depends condition not met)" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.473Z" level=info msg="TaskSet Reconciliation" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.473Z" level=info msg=reconcileAgentPod namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:36.481Z" level=info msg="Workflow update successful" namespace=core-support phase=Running resourceVersion=43611567 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="Processing workflow" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="Task-result reconciliation" namespace=core-support numObjs=0 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="node changed" namespace=core-support new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-to-retry-6jqss-2496561594 old.message= old.phase=Pending old.progress=0/1 workflow
=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="Pod failed: Error (exit code 2)" displayName=continue namespace=core-support pod=dag-to-retry-6jqss-step-3259869403 templateName=step workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="node unchanged" namespace=core-support nodeID=dag-to-retry-6jqss-3259869403 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="node unchanged" namespace=core-support nodeID=dag-to-retry-6jqss-1212128117 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="Pod failed: Error (exit code 1)" displayName=failure namespace=core-support pod=dag-to-retry-6jqss-step-3193705776 templateName=step workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="node unchanged" namespace=core-support nodeID=dag-to-retry-6jqss-3193705776 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="Outbound nodes of dag-to-retry-6jqss set to [dag-to-retry-6jqss-2496561594 dag-to-retry-6jqss-3806205299]" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.474Z" level=info msg="node dag-to-retry-6jqss phase Running -> Failed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="node dag-to-retry-6jqss finished: 2023-07-19 16:30:46.47500318 +0000 UTC" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="Checking daemoned children of dag-to-retry-6jqss" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="TaskSet Reconciliation" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg=reconcileAgentPod namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="Updated phase Running -> Failed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="Marking workflow completed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="Doesn't match with archive label selector. Skipping Archive" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.475Z" level=info msg="Checking daemoned children of " namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.480Z" level=info msg="cleaning up pod" action=deletePod key=core-support/dag-to-retry-6jqss-1340600742-agent/deletePod
time="2023-07-19T16:30:46.484Z" level=info msg="Workflow update successful" namespace=core-support phase=Failed resourceVersion=43611870 workflow=dag-to-retry-6jqss
time="2023-07-19T16:30:46.485Z" level=info msg="Queueing Failed workflow core-support/dag-to-retry-6jqss for delete in 168h0m0s due to TTL"
time="2023-07-19T16:30:46.495Z" level=info msg="cleaning up pod" action=labelPodCompleted key=core-support/dag-to-retry-6jqss-step-3193705776/labelPodCompleted
time="2023-07-19T16:30:46.495Z" level=info msg="cleaning up pod" action=labelPodCompleted key=core-support/dag-to-retry-6jqss-step-2496561594/labelPodCompleted
time="2023-07-19T16:30:46.495Z" level=info msg="cleaning up pod" action=labelPodCompleted key=core-support/dag-to-retry-6jqss-step-1212128117/labelPodCompleted
time="2023-07-19T16:30:46.495Z" level=info msg="cleaning up pod" action=labelPodCompleted key=core-support/dag-to-retry-6jqss-step-3259869403/labelPodCompleted
time="2023-07-19T16:31:53.513Z" level=info msg="Processing workflow" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Task-result reconciliation" namespace=core-support numObjs=0 workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=error msg="Failed to build local scope from task" namespace=core-support taskName=task-after-continue workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=error msg="Mark error node" error="Ancestor task node continue not found" namespace=core-support nodeName=dag-to-retry-6jqss.task-after-continue workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=error msg="node is already fulfilled" fromPhase=Succeeded namespace=core-support nodeName=dag-to-retry-6jqss.task-after-continue toPhase=Error workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="node dag-to-retry-6jqss-2496561594 phase Succeeded -> Error" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="node dag-to-retry-6jqss-2496561594 message: Ancestor task node continue not found" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=error msg="Mark error node" error="Ancestor task node continue not found" namespace=core-support nodeName=dag-to-retry-6jqss workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="node dag-to-retry-6jqss phase Running -> Error" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="node dag-to-retry-6jqss message: Ancestor task node continue not found" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="node dag-to-retry-6jqss finished: 2023-07-19 16:31:53.513785253 +0000 UTC" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Checking daemoned children of dag-to-retry-6jqss" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=error msg="Mark error node" error="Ancestor task node continue not found" namespace=core-support nodeName=dag-to-retry-6jqss workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=error msg="error in entry template execution" error="Ancestor task node continue not found" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Updated phase Running -> Error" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Updated message  -> error in entry template execution: Ancestor task node continue not found" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Marking workflow completed" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Doesn't match with archive label selector. Skipping Archive" namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.513Z" level=info msg="Checking daemoned children of " namespace=core-support workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.519Z" level=info msg="cleaning up pod" action=deletePod key=core-support/dag-to-retry-6jqss-1340600742-agent/deletePod
time="2023-07-19T16:31:53.523Z" level=info msg="Workflow update successful" namespace=core-support phase=Error resourceVersion=43613472 workflow=dag-to-retry-6jqss
time="2023-07-19T16:31:53.523Z" level=info msg="Queueing Error workflow core-support/dag-to-retry-6jqss for delete in 168h0m0s due to TTL"

Logs from in your workflow's wait container

$ kubectl logs -c wait -l workflows.argoproj.io/workflow=dag-to-retry-6jqss,workflow.argoproj.io/phase!=Succeeded
time="2023-07-19T16:30:18.065Z" level=info msg="Starting Workflow Executor" version=v3.4.7
time="2023-07-19T16:30:18.067Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-07-19T16:30:18.067Z" level=info msg="Executor initialized" deadline="2023-07-19 17:30:16 +0000 UTC" includeScriptOutput=false namespace=core-support podName=dag-to-retry-6jqss-step-1212128117 template="{\"name\":\"ste
p\",\"inputs\":{\"parameters\":[{\"name\":\"exitCode\",\"value\":\"0\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"alpine:3.7\",\"command\":[\"sh\",\"-c\",\"exit 0\"],\"resources\":{}}}" version="&
Version{Version:v3.4.7,BuildDate:2023-04-11T16:19:29Z,GitCommit:f2292647c5a6be2f888447a1fef71445cc05b8fd,GitTag:v3.4.7,GitTreeState:clean,GoVersion:go1.19.8,Compiler:gc,Platform:linux/amd64,}"
time="2023-07-19T16:30:18.067Z" level=info msg="Starting deadline monitor"
time="2023-07-19T16:30:21.068Z" level=info msg="Main container completed" error="<nil>"
time="2023-07-19T16:30:21.068Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-07-19T16:30:21.068Z" level=info msg="No output parameters"
time="2023-07-19T16:30:21.068Z" level=info msg="No output artifacts"
time="2023-07-19T16:30:21.068Z" level=info msg="Alloc=7928 TotalAlloc=14995 Sys=30573 NumGC=5 Goroutines=7"
time="2023-07-19T16:30:21.068Z" level=info msg="Deadline monitor stopped"
time="2023-07-19T16:30:38.082Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-07-19T16:30:38.082Z" level=info msg="Executor initialized" deadline="2023-07-19 17:30:16 +0000 UTC" includeScriptOutput=false namespace=core-support podName=dag-to-retry-6jqss-step-2496561594 template="{\"name\":\"ste
p\",\"inputs\":{\"parameters\":[{\"name\":\"exitCode\",\"value\":\"0\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"alpine:3.7\",\"command\":[\"sh\",\"-c\",\"exit 0\"],\"resources\":{}}}" version="&
Version{Version:v3.4.7,BuildDate:2023-04-11T16:19:29Z,GitCommit:f2292647c5a6be2f888447a1fef71445cc05b8fd,GitTag:v3.4.7,GitTreeState:clean,GoVersion:go1.19.8,Compiler:gc,Platform:linux/amd64,}"
time="2023-07-19T16:30:38.082Z" level=info msg="Starting deadline monitor"
time="2023-07-19T16:30:40.082Z" level=info msg="Main container completed" error="<nil>"
time="2023-07-19T16:30:40.083Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-07-19T16:30:40.083Z" level=info msg="No output parameters"
time="2023-07-19T16:30:40.083Z" level=info msg="No output artifacts"
time="2023-07-19T16:30:40.083Z" level=info msg="Alloc=7155 TotalAlloc=15016 Sys=30829 NumGC=5 Goroutines=7"
time="2023-07-19T16:30:40.083Z" level=info msg="Deadline monitor stopped"
time="2023-07-19T16:30:40.083Z" level=info msg="stopping progress monitor (context done)" error="context canceled"

[JPZ13]

Hey @terrytangyuan - this seems somewhat adjacent to your work on #9141. Do you have any advice on the fix, or would you like to work on it?

[terrytangyuan]

There might be a bug in the retry logic here that causes certain nodes to be accidentally removed: https://github.com/argoproj/argo-workflows/blob/master/workflow/util/util.go#L804

[shuangkun]

Reproduce it and will find the root cause to fix.