workflow status node output parameters is missing occasionally when use withParam

[ChenRussell]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

workflow status node output parameters is missing occasionally, and it's hard to reproduce it.
I checked the wait container exit code is 0, and the main container logs are successfully uploaded to s3, so I think wait container has successfully created the workflowtaskresult.
My pod gc strategy is OnPodCompletion, when pod was deleted, the workflowtaskresult will be deleted too. I think the problem is in the workflow-controller, I have checked the related code, but can't find the root cause.

The correct node status output is:

The abnormal node status output is like this:

Version(s)

v3.4.8

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: parallel-tasks-
spec:
  entrypoint: flow
  templates:
    - name: flow
      dag:
        tasks:
          - name: start
            template: argosay
            arguments: {}
          - name: start-test-output
            template: argosay
            dependencies:
              - start
          - name: start-test-output-foreach-input-list
            template: start-test-output-foreach-input-list
            arguments:
              parameters:
                - name: split-index
                  value: "{{item}}"
            dependencies:
              - start-test-output
            withParam: "[0,1,2,3,4,5,6,7,8,9]"
          - name: end-test-output
            template: argosay
            dependencies:
              - start-test-output-foreach-input-list

    - name: start-test-output-foreach-input-list
      inputs:
        parameters:
          - name: split-index
      outputs:
        parameters:
          - name: task-id
            valueFrom:
              parameter: '{{tasks.test-output.outputs.parameters.task-id}}'
      dag:
        tasks:
          - name: test-output
            template: test-output
            arguments:
              parameters:
                - name: task-index
                  value: "{{inputs.parameters.split-index}}"
        failFast: true

    - name: argosay
      script:
        image: python:3.6-alpine
        command: [python]
        source: |
          import random
          i = random.randint(1, 100)
          print(i)

    - name: test-output
      inputs:
        parameters:
          - name: task-index
      outputs:
        parameters:
          - name: task-id
            valueFrom:
              path: /mnt/out/task_id
      activeDeadlineSeconds: 3600
      retryStrategy:
        limit: 1
        retryPolicy: Always
        backoff:
          duration: "10s"
        failFast: true
      container:
        image: python:3.6-alpine
        command: [sh, -c]
        args: ["echo -n hello-{{inputs.parameters.task-index}} > /mnt/out/task_id"]
        volumeMounts:
          - name: out
            mountPath: /mnt/out
      volumes:
        - name: out
          emptyDir: {}

Logs from the workflow controller

nil

Logs from in your workflow's wait container

nil

[chengjoey]

@ChenRussell have you tested in latest, I tried 5 times and couldn't reproduce it.

[ChenRussell]

@ChenRussell have you tested in latest, I tried 5 times and couldn't reproduce it.

It's hard to reproduce it, but it exists in prod env, I can't reproduce it in test env either.

[isubasinghe]

@ChenRussell would you be able to confirm if a patch I make fixes your issue ?
I think I know how to fix this.

[ChenRussell]

@ChenRussell would you be able to confirm if a patch I make fixes your issue ? I think I know how to fix this.

Yes, that's great, I’ll test it to confirm if it resolves the issue.