Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Every loop of an inline template receives the first item #12594

Closed
3 of 4 tasks
ezequiel-hemisphere opened this issue Jan 31, 2024 · 0 comments · Fixed by #12628
Closed
3 of 4 tasks

Every loop of an inline template receives the first item #12594

ezequiel-hemisphere opened this issue Jan 31, 2024 · 0 comments · Fixed by #12628
Assignees
Labels
area/looping `withParams`, `withItems`, and `withSequence` area/workflow-templates P3 Low priority solution/workaround There's a workaround, might not be great, but exists type/bug

Comments

@ezequiel-hemisphere
Copy link

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issue exists when I tested with :latest
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

This only happens when a workflow calls a WorkflowTemplate's template, and this template has:

  • An iterated template (withParam or withItems)
  • The template that gets iterated itself is a DAG/Steps with an inline task/step

Each instance of the task/step will receive the same value of the item (even though the UI shows the name of the instances with the different item values).

If the step/task is made to reference a named template, the problem goes away.

In the example workflow, note how step bar get the arg bar, but outputs food both as an output argument as well as what it prints in the logs.

Version

v3.5.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-inline-iterated
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: iterated
            template: steps-inline
            arguments:
              parameters:
                - name: arg
                  value: "{{ item }}"
            withItems:
              - foo
              - bar

    - name: steps-inline
      inputs:
        parameters:
          - name: arg
      steps:
        - - name: inline
            arguments:
              parameters:
                - name: arg
                  value: "{{ inputs.parameters.arg }}"
            inline:
              inputs:
                parameters:
                  - name: arg
              container:
                image: docker/whalesay
                command: [echo]
                args:
                  - "{{ inputs.parameters.arg }}"
              outputs:
                parameters:
                  - name: arg-out
                    value: "{{ inputs.parameters.arg }}"
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: test-call-inline-iterated
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: process
            templateRef:
              name: test-inline-iterated
              template: main

Logs from the workflow controller

time="2024-01-31T04:19:47.975Z" level=info msg="Processing workflow" Phase= ResourceVersion=851105342 namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=info msg="DAG node test-call-inline-iterated initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=warning msg="was unable to obtain the node for test-call-inline-iterated-977946544, taskName process"
time="2024-01-31T04:19:48.003Z" level=warning msg="was unable to obtain the node for test-call-inline-iterated-977946544, taskName process"
time="2024-01-31T04:19:48.003Z" level=info msg="All of node test-call-inline-iterated.process dependencies [] completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.003Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.004Z" level=info msg="Steps node test-call-inline-iterated-977946544 initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.004Z" level=info msg="StepGroup node test-call-inline-iterated-632750730 initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.004Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.004Z" level=info msg="Steps node test-call-inline-iterated-2277014655 initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.004Z" level=info msg="StepGroup node test-call-inline-iterated-3607770987 initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.005Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.005Z" level=info msg="Pod node test-call-inline-iterated-314658614 initialized Pending" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.082Z" level=info msg="Created pod: test-call-inline-iterated.process[0].iterated(0:foo)[0].inline (test-call-inline-iterated-314658614)" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.083Z" level=info msg="Workflow step group node test-call-inline-iterated-3607770987 not yet completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.083Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.083Z" level=info msg="Steps node test-call-inline-iterated-1829332701 initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.083Z" level=info msg="StepGroup node test-call-inline-iterated-3378349025 initialized Running" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.084Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.084Z" level=info msg="Pod node test-call-inline-iterated-2316520508 initialized Pending" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.156Z" level=info msg="Created pod: test-call-inline-iterated.process[0].iterated(1:bar)[0].inline (test-call-inline-iterated-2316520508)" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.156Z" level=info msg="Workflow step group node test-call-inline-iterated-3378349025 not yet completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.156Z" level=info msg="Workflow step group node test-call-inline-iterated-632750730 not yet completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.156Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.156Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:48.181Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=851105351 workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.083Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=851105351 namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.084Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.084Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.085Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.085Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=test-call-inline-iterated-2316520508 old.message= old.phase=Pending old.progress=0/1 workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.085Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.085Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.085Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-call-inline-iterated-314658614 old.message= old.phase=Pending old.progress=0/1 workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="Step group node test-call-inline-iterated-3607770987 successful" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="node test-call-inline-iterated-3607770987 phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="node test-call-inline-iterated-3607770987 finished: 2024-01-31 04:19:58.086079565 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="Outbound nodes of test-call-inline-iterated-314658614 is [test-call-inline-iterated-314658614]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="Outbound nodes of test-call-inline-iterated-2277014655 is [test-call-inline-iterated-314658614]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="node test-call-inline-iterated-2277014655 phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.086Z" level=info msg="node test-call-inline-iterated-2277014655 finished: 2024-01-31 04:19:58.086292239 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.087Z" level=info msg="Workflow step group node test-call-inline-iterated-3378349025 not yet completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.087Z" level=info msg="Workflow step group node test-call-inline-iterated-632750730 not yet completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.087Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.087Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.090Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/test-call-inline-iterated-2316520508/terminateContainers
time="2024-01-31T04:19:58.100Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=851105463 workflow=test-call-inline-iterated
time="2024-01-31T04:19:58.107Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/test-call-inline-iterated-314658614/labelPodCompleted
time="2024-01-31T04:20:08.139Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=851105463 namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.140Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.140Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.140Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.140Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-call-inline-iterated-2316520508 old.message= old.phase=Running old.progress=0/1 workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="Step group node test-call-inline-iterated-3378349025 successful" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="node test-call-inline-iterated-3378349025 phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="node test-call-inline-iterated-3378349025 finished: 2024-01-31 04:20:08.142210934 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="Outbound nodes of test-call-inline-iterated-2316520508 is [test-call-inline-iterated-2316520508]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="Outbound nodes of test-call-inline-iterated-1829332701 is [test-call-inline-iterated-2316520508]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="node test-call-inline-iterated-1829332701 phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="node test-call-inline-iterated-1829332701 finished: 2024-01-31 04:20:08.142503083 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="Step group node test-call-inline-iterated-632750730 successful" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="node test-call-inline-iterated-632750730 phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="node test-call-inline-iterated-632750730 finished: 2024-01-31 04:20:08.142568045 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.142Z" level=info msg="Outbound nodes of test-call-inline-iterated-2277014655 is [test-call-inline-iterated-314658614]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="Outbound nodes of test-call-inline-iterated-1829332701 is [test-call-inline-iterated-2316520508]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="Outbound nodes of test-call-inline-iterated-977946544 is [test-call-inline-iterated-314658614 test-call-inline-iterated-2316520508]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="node test-call-inline-iterated-977946544 phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="node test-call-inline-iterated-977946544 finished: 2024-01-31 04:20:08.143089854 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="Outbound nodes of test-call-inline-iterated set to [test-call-inline-iterated-314658614 test-call-inline-iterated-2316520508]" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="node test-call-inline-iterated phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="node test-call-inline-iterated finished: 2024-01-31 04:20:08.143555416 +0000 UTC" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="Marking workflow completed" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.143Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.148Z" level=info msg="cleaning up pod" action=deletePod key=argo/test-call-inline-iterated-1340600742-agent/deletePod
time="2024-01-31T04:20:08.158Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=851105554 workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.162Z" level=info msg="archiving workflow" namespace=argo uid=777dc246-e96b-45d2-b0c7-5d7c3ed23a35 workflow=test-call-inline-iterated
time="2024-01-31T04:20:08.167Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/test-call-inline-iterated-2316520508/labelPodCompleted
time="2024-01-31T04:20:08.189Z" level=info msg="Queueing Succeeded workflow argo/test-call-inline-iterated for delete in 72h0m0s due to TTL"
time="2024-01-31T04:20:28.091Z" level=info msg="cleaning up pod" action=killContainers key=argo/test-call-inline-iterated-2316520508/killContainers

Logs from in your workflow's wait container

time="2024-01-31T04:19:56.055Z" level=info msg="No output artifacts"
time="2024-01-31T04:19:56.055Z" level=info msg="GCS Save path: /tmp/argo/outputs/logs/main.log, key: test-call-inline-iterated/test-call-inline-iterated-2316520508/main.log"
time="2024-01-31T04:19:56.930Z" level=info msg="Save artifact" artifactName=main-logs duration=874.475134ms error="<nil>" key=test-call-inline-iterated/test-call-inline-iterated-2316520508/main.log
time="2024-01-31T04:19:56.930Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-01-31T04:19:56.930Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-01-31T04:19:56.987Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-workflow-sa\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-01-31T04:19:57.092Z" level=info msg="Alloc=26376 TotalAlloc=33025 Sys=39525 NumGC=5 Goroutines=11"
time="2024-01-31T04:19:57.094Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" error="workflowtaskresults.argoproj.io \"test-call-inline-iterated-2316520508\" is forbidden: User \"system:serviceaccount:argo:argo-workflow-sa\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-01-31T04:19:57.123Z" level=info msg="Deadline monitor stopped"
time="2024-01-31T04:19:57.123Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-01-31T04:19:54.406Z" level=info msg="Saving output parameters"
time="2024-01-31T04:19:54.407Z" level=info msg="Saving path output parameter: arg-out"
time="2024-01-31T04:19:54.407Z" level=info msg="No output artifacts"
time="2024-01-31T04:19:54.408Z" level=info msg="GCS Save path: /tmp/argo/outputs/logs/main.log, key: test-call-inline-iterated/test-call-inline-iterated-314658614/main.log"
time="2024-01-31T04:19:55.006Z" level=info msg="Save artifact" artifactName=main-logs duration=598.129898ms error="<nil>" key=test-call-inline-iterated/test-call-inline-iterated-314658614/main.log
time="2024-01-31T04:19:55.006Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-01-31T04:19:55.006Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-01-31T04:19:55.008Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-workflow-sa\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-01-31T04:19:55.039Z" level=info msg="Alloc=26373 TotalAlloc=33027 Sys=39525 NumGC=5 Goroutines=11"
time="2024-01-31T04:19:55.041Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" error="workflowtaskresults.argoproj.io \"test-call-inline-iterated-314658614\" is forbidden: User \"system:serviceaccount:argo:argo-workflow-sa\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
@agilgur5 agilgur5 added area/workflow-templates area/looping `withParams`, `withItems`, and `withSequence` P3 Low priority solution/workaround There's a workaround, might not be great, but exists labels Jan 31, 2024
@shuangkun shuangkun self-assigned this Feb 6, 2024
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue Feb 6, 2024
juliev0 pushed a commit that referenced this issue Mar 15, 2024
…ixes: #12594 (#12628)

Signed-off-by: shuangkun <tsk2013uestc@163.com>
agilgur5 pushed a commit that referenced this issue Apr 19, 2024
…ixes: #12594 (#12628)

Signed-off-by: shuangkun <tsk2013uestc@163.com>
(cherry picked from commit d5a4f7e)
@agilgur5 agilgur5 added this to the v3.5.x patches milestone Apr 19, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue May 6, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/looping `withParams`, `withItems`, and `withSequence` area/workflow-templates P3 Low priority solution/workaround There's a workaround, might not be great, but exists type/bug
Projects
None yet
3 participants