Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A memoized steps hit, the following nodes not displayed in the UI #7873

Closed
zjgemi opened this issue Feb 13, 2022 · 2 comments · Fixed by #12780
Closed

A memoized steps hit, the following nodes not displayed in the UI #7873

zjgemi opened this issue Feb 13, 2022 · 2 comments · Fixed by #12780
Assignees
Labels
area/memoization solution/workaround There's a workaround, might not be great, but exists type/bug

Comments

@zjgemi
Copy link
Contributor

zjgemi commented Feb 13, 2022

Summary

When a memoized step with steps template is hit by a ConfigMap record, the nodes following this step are not displayed in the UI (despite they run expectedly). The behavior might result from that the memoized step has no children.

Diagnostics

The smallest workflow that reproduces the bug looks like

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: memoized-bug-
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: hello-steps
        template: memoized
    - - name: whatever
        template: hello

  - name: memoized
    outputs:
      parameters:
      - name: msg
        valueFrom:
          parameter: "{{steps.hello-step.outputs.result}}"
    steps:
    - - name: hello-step
        template: hello
    memoize:
      key: "memoized-bug-steps-0"
      cache:
        configMap:
          name: my-config

  - name: hello
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo Hello"]

When the workflow is submitted for the first time, the step hello-steps is stored in the ConfigMap. And in the UI, the workflow looks like
workflow-1
Everything is OK. But when the workflow is submitted for the second time, the step hello-steps is loaded from the ConfigMap, the following step whatever is not displayed in the UI
workflow-2

In the detail of the second workflow

$ argo get memoized-bug-nmvgq -n argo -o yaml
...
    memoized-bug-nmvgq-667942788:
      boundaryID: memoized-bug-nmvgq
      displayName: hello-steps
      finishedAt: "2022-02-13T05:32:52Z"
      id: memoized-bug-nmvgq-667942788
      memoizationStatus:
        cacheName: my-config
        hit: true
        key: memoized-bug-steps-0
      name: memoized-bug-nmvgq[0].hello-steps
      outputs:
        parameters:
        - name: msg
          value: Hello
      phase: Succeeded
      startedAt: "2022-02-13T05:32:52Z"
      templateName: memoized
      templateScope: local/memoized-bug-nmvgq
      type: Steps
...
    memoized-bug-nmvgq-2677016171:
      boundaryID: memoized-bug-nmvgq
      displayName: whatever
      finishedAt: "2022-02-13T05:33:05Z"
      hostNodeName: minikube
      id: memoized-bug-nmvgq-2677016171
      name: memoized-bug-nmvgq[1].whatever
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: memoized-bug-nmvgq/memoized-bug-nmvgq-2677016171/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 2
      startedAt: "2022-02-13T05:32:52Z"
      templateName: hello
      templateScope: local/memoized-bug-nmvgq
      type: Pod

The step hello-steps does not have children which may cause the step whatever not displayed in the UI, despite the step whatever succeeded.


Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

@sarabala1979
Copy link
Member

@zjgemi memoization is only storing the node status of a particular step/task, not an original template. I don't think we can get memoization templates.
one workaround: instead of configuring memoization on definition templates like steps/dag, you can configure memoization on execution templates like container, script, etc. It will give the same output.

@sarabala1979 sarabala1979 added area/cron-workflows solution/workaround There's a workaround, might not be great, but exists area/memoization and removed triage area/cron-workflows labels Feb 14, 2022
@zjgemi
Copy link
Contributor Author

zjgemi commented Feb 14, 2022

@zjgemi memoization is only storing the node status of a particular step/task, not an original template. I don't think we can get memoization templates.
one workaround: instead of configuring memoization on definition templates like steps/dag, you can configure memoization on execution templates like container, script, etc. It will give the same output.

I understand the workaround. Thanks a lot.

zjgemi referenced this issue in deepmodeling/argo-workflows Mar 10, 2024
Signed-off-by: zjgemi <liuxin_zijian@163.com>
isubasinghe added a commit that referenced this issue Mar 22, 2024
…AG. Fixes: #7873 (#12780)

Signed-off-by: shuangkun <tsk2013uestc@163.com>
Co-authored-by: zjgemi <liuxin_zijian@163.com>
Co-authored-by: sherwinkoo29 <sherwinkoo@163.com>
Co-authored-by: Isitha Subasinghe <isitha@pipekit.io>
@shuangkun shuangkun self-assigned this Mar 22, 2024
@agilgur5 agilgur5 added this to the v3.5.x patches milestone Apr 19, 2024
agilgur5 pushed a commit that referenced this issue Apr 19, 2024
…AG. Fixes: #7873 (#12780)

Signed-off-by: shuangkun <tsk2013uestc@163.com>
Co-authored-by: zjgemi <liuxin_zijian@163.com>
Co-authored-by: sherwinkoo29 <sherwinkoo@163.com>
Co-authored-by: Isitha Subasinghe <isitha@pipekit.io>
(cherry picked from commit a719d94)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/memoization solution/workaround There's a workaround, might not be great, but exists type/bug
Projects
None yet
4 participants