Skip to content

Inconsistent behavior for Workflow Arguments vs Template inputs to prioritise Value over ConfigMapKeyRef #14426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 of 4 tasks
elliotgunton opened this issue Apr 25, 2025 · 4 comments · May be fixed by #14462
Open
3 of 4 tasks

Comments

@elliotgunton
Copy link
Contributor

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Same issue as #10182 but for Workflow.Spec.Arguments (and referenced via {{workflow.parameters.param-name}}) which are dealt with in a different code path here:

if param.ValueFrom != nil && param.ValueFrom.ConfigMapKeyRef != nil {
cmValue, err := common.GetConfigMapValue(woc.controller.configMapInformer.GetIndexer(), woc.wf.Namespace, param.ValueFrom.ConfigMapKeyRef.Name, param.ValueFrom.ConfigMapKeyRef.Key)
if err != nil {
if param.ValueFrom.Default != nil {
woc.globalParams["workflow.parameters."+param.Name] = param.ValueFrom.Default.String()
} else {
return fmt.Errorf("failed to set global parameter %s from configmap with name %s and key %s: %w",
param.Name, param.ValueFrom.ConfigMapKeyRef.Name, param.ValueFrom.ConfigMapKeyRef.Key, err)
}
} else {
woc.globalParams["workflow.parameters."+param.Name] = cmValue
}
} else if param.Value != nil {
woc.globalParams["workflow.parameters."+param.Name] = param.Value.String()

Which means passing a value to an argument that uses a ValueFrom.ConfigMapKeyRef does nothing, when it should prioritise the value to match template input behaviour. A workaround is moving the config map input to a template to get the fixed behaviour from #10182.

Version(s)

v3.6.5

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-properties
  namespace: argo
  labels:
    workflows.argoproj.io/configmap-type: Parameter
data:
  message: "message from config map"
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: wf-template-echo
  namespace: argo
spec:
  entrypoint: echo
  arguments:
    parameters:
      - name: message
        valueFrom:
          configMapKeyRef:
            name: config-properties
            key: message
  templates:
    - name: echo
      container:
        image: busybox
        command: [echo]
        args: ["{{workflow.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: wf-parameter-overwrite-
  namespace: argo
spec:
  entrypoint: echo
  arguments:
    parameters:
      - name: message
        value: "configmap argument overwrite with argument"
  workflowTemplateRef:
    name: wf-template-echo

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2025-04-24T14:04:56.898Z" level=info msg="Processing workflow" Phase= ResourceVersion=864971 namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.906Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.906Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.906Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.906Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.906Z" level=info msg="Pod node wf-parameter-overwrite-8z8bv initialized Pending" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.907Z" level=warning msg="couldn't get boundaryTemplate through nodeName wf-parameter-overwrite-8z8bv" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.916Z" level=info msg="Created pod: wf-parameter-overwrite-8z8bv (wf-parameter-overwrite-8z8bv)" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.916Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.916Z" level=info msg=reconcileAgentPod namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56Z" level=info msg="add pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.922Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=864975 workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.922Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=864975 namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.922Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.922Z" level=info msg="node unchanged" namespace=argo nodeID=wf-parameter-overwrite-8z8bv workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.922Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56.922Z" level=info msg=reconcileAgentPod namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:56Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:04:58Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:00Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:01Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:02Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:03Z" level=info msg="update pod event" pod=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.933Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=864975 namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.939Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.940Z" level=info msg="task-result changed" namespace=argo nodeID=wf-parameter-overwrite-8z8bv workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.941Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=wf-parameter-overwrite-8z8bv old.message= old.phase=Pending old.progress=0/1 workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.943Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.943Z" level=info msg=reconcileAgentPod namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.943Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.944Z" level=info msg="Marking workflow completed" namespace=argo workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06.959Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=865012 workflow=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=argo podName=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:06Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/wf-parameter-overwrite-8z8bv/labelPodCompleted namespace=argo podName=wf-parameter-overwrite-8z8bv
time="2025-04-24T14:05:07Z" level=info msg="delete pod event" pod=wf-parameter-overwrite-8z8bv

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
time="2025-04-24T14:05:01.152Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-04-24T14:05:01.152Z" level=info msg="No output parameters"
time="2025-04-24T14:05:01.152Z" level=info msg="No output artifacts"
time="2025-04-24T14:05:01.154Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: wf-parameter-overwrite-8z8bv/wf-parameter-overwrite-8z8bv/main.log"
time="2025-04-24T14:05:01.154Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2025-04-24T14:05:01.154Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=wf-parameter-overwrite-8z8bv/wf-parameter-overwrite-8z8bv/main.log path=/tmp/argo/outputs/logs/main.log
time="2025-04-24T14:05:01.178Z" level=info msg="Save artifact" artifactName=main-logs duration=23.951542ms error="<nil>" key=wf-parameter-overwrite-8z8bv/wf-parameter-overwrite-8z8bv/main.log
time="2025-04-24T14:05:01.178Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2025-04-24T14:05:01.178Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2025-04-24T14:05:01.208Z" level=info msg="Alloc=11599 TotalAlloc=17022 Sys=23381 NumGC=4 Goroutines=10"
@nicdard
Copy link

nicdard commented Apr 28, 2025

Hello, I would like to contribute this fix :)

@Joibel
Copy link
Member

Joibel commented Apr 29, 2025

Yes please @nicdard

nicdard added a commit to nicdard/argo-workflows that referenced this issue May 8, 2025
Signed-off-by: Nicola Dardanis <nicdard@gmail.com>
@nicdard nicdard linked a pull request May 8, 2025 that will close this issue
@MasonM
Copy link
Member

MasonM commented May 11, 2025

Thanks for the report! Here's a slightly simpler workflow to reproduce this discrepancy that doesn't use a WorkflowTemplate:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-properties
  labels:
    workflows.argoproj.io/configmap-type: Parameter
data:
  message: "message from config map"
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: parameter-valuefrom-value-
spec:
  entrypoint: echo
  arguments: &args
    parameters:
      - name: message
        value: "configmap argument overwrite with argument"
        valueFrom:
          configMapKeyRef:
            name: config-properties
            key: message
  templates:
    - name: echo
      inputs: *args
      container:
        image: busybox
        args: ["printf", "WORKFLOW: %s\nINPUT: %s", "{{workflow.parameters.message}}", "{{inputs.parameters.message}}"]

Output when submitting this using ./dist/argo submit --log valuebug.yaml :

Name:                parameter-valuefrom-value-7mb47
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Pending
Created:             Sun May 11 22:05:18 +0000 (now)
Progress:            
Parameters:          
  message:           configmap argument overwrite with argument
parameter-valuefrom-value-7mb47: time="2025-05-11T22:05:31.719Z" level=info msg="capturing logs" argo=true
parameter-valuefrom-value-7mb47: WORKFLOW: message from config map
parameter-valuefrom-value-7mb47: time="2025-05-11T22:05:31.720Z" level=debug msg="ignore signal child exited" argo=true
parameter-valuefrom-value-7mb47: time="2025-05-11T22:05:32.720Z" level=info msg="sub-process exited" argo=true error="<nil>"
parameter-valuefrom-value-7mb47: INPUT: configmap argument overwrite with argument

@nicdard
Copy link

nicdard commented May 11, 2025

Thanks for the report! Here's a slightly simpler workflow to reproduce this discrepancy that doesn't use a WorkflowTemplate:

apiVersion: v1
kind: ConfigMap
metadata:
name: config-properties
labels:
workflows.argoproj.io/configmap-type: Parameter
data:
message: "message from config map"

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: parameter-valuefrom-value-
spec:
entrypoint: echo
arguments: &args
parameters:
- name: message
value: "configmap argument overwrite with argument"
valueFrom:
configMapKeyRef:
name: config-properties
key: message
templates:
- name: echo
inputs: *args
container:
image: busybox
args: ["printf", "WORKFLOW: %s\nINPUT: %s", "{{workflow.parameters.message}}", "{{inputs.parameters.message}}"]
Output when submitting this using ./dist/argo submit --log valuebug.yaml :

Name:                parameter-valuefrom-value-7mb47
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Pending
Created:             Sun May 11 22:05:18 +0000 (now)
Progress:            
Parameters:          
  message:           configmap argument overwrite with argument
parameter-valuefrom-value-7mb47: time="2025-05-11T22:05:31.719Z" level=info msg="capturing logs" argo=true
parameter-valuefrom-value-7mb47: WORKFLOW: message from config map
parameter-valuefrom-value-7mb47: time="2025-05-11T22:05:31.720Z" level=debug msg="ignore signal child exited" argo=true
parameter-valuefrom-value-7mb47: time="2025-05-11T22:05:32.720Z" level=info msg="sub-process exited" argo=true error="<nil>"
parameter-valuefrom-value-7mb47: INPUT: configmap argument overwrite with argument

Hey @MasonM thanks this is very helpful!

nicdard added a commit to nicdard/argo-workflows that referenced this issue May 12, 2025
Signed-off-by: Nicola Dardanis <nicdard@gmail.com>
nicdard added a commit to nicdard/argo-workflows that referenced this issue May 12, 2025
nicdard added a commit to nicdard/argo-workflows that referenced this issue May 12, 2025
…roj#14426

Signed-off-by: Nicola Dardanis <nicdard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants