feat: support multiple SSM parameters for large runner matcher configs (#4790) #4792
Conversation
trigger ci with a minor change
There was a problem hiding this comment.
@edersonbrilhante nice contribution. I just tried the parameter split with the multi-runner example in this repo. By adding a few config and adding more labes. Result is I got this error, related to the added foreach
│
│ on ../../modules/webhook/webhook.tf line 27, in resource "aws_ssm_parameter" "runner_matcher_config":
│ 27: for_each = { for idx, val in local.matcher_chunks : idx => val }
│ ├────────────────
│ │ local.matcher_chunks will be known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will
│ identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully
│ converge.
Wondering you you tested the change? I was not able to get it working right now.
|
I did test it, but I forgot to include a large configuration and only tested the normal case. My bad. I found the same bug now. |
|
@npalm I just pushed my fix. I tested a normal config and a tested a really big one with 7 ssm resources and it worked fine now. |
|
|
||
| resource "aws_ssm_parameter" "runner_matcher_config" { | ||
| for_each = { for idx, val in local.matcher_chunks : idx => val } | ||
| count = length(local.matcher_chunks) |
There was a problem hiding this comment.
This count still results in my case in the following error with a large configuration:
Plan: 118 to add, 7 to change, 0 to destroy.
╷
│ Error: Invalid count argument
│
│ on ../../modules/webhook/webhook.tf line 27, in resource "aws_ssm_parameter" "runner_matcher_config":
│ 27: count = length(local.matcher_chunks)
│
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around
│ this, use the -target argument to first apply only the resources that the count depends on.
There was a problem hiding this comment.
@edersonbrilhante just tested with the example multi-runner in the repo. With about 4 extra config files with quite some labels. But ending up with the error above.
PR Description
Implements support for splitting the runner matcher configuration across multiple SSM parameters.
PARAMETER_RUNNER_MATCHER_CONFIG_PATHcan now accept multiple parameter paths separated by a colon (:).This avoids SSM size limits for large configurations and improves scalability for environments with many runner types and labels.
Closes #4790