Skip to content

env: add new extended format to handle environment variables #10862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
+965 −115
Open

env: add new extended format to handle environment variables #10862

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
75c8668
config_format: allow list and object in env section
edsiper Sep 9, 2025
0f9bb7f
config_map: new flag to indicate that value can be mutable/dynamic
edsiper Sep 9, 2025
9224e6d
config: use new flb_env_set_extended()
edsiper Sep 9, 2025
f161887
custom: adjust api for env translation
edsiper Sep 9, 2025
c085436
input: store raw property value, configmap translate it later
edsiper Sep 9, 2025
4654fed
reload: on reconstructing use new config map env prototype
edsiper Sep 9, 2025
d5948fe
output: use new config_map_set prototype
edsiper Sep 9, 2025
d78f10f
filter: use new config_map_set prototype
edsiper Sep 9, 2025
4b71063
processor: use new config_map_set prototype
edsiper Sep 9, 2025
746b463
env: add support for extended environment variables
edsiper Sep 9, 2025
8b4af90
processor_sampling: use new config_map_set prototype
edsiper Sep 9, 2025
fcb9bfe
in_dummy: allow dynamic values in body and metadata
edsiper Sep 9, 2025
c4f30bf
tests: internal: config_map: use new prototype function
edsiper Sep 9, 2025
f33d942
tests: internal: fuzzers: config_map: use new func prototype
edsiper Sep 9, 2025
ee16927
tests: internal: env: add unit tests for extended env format
edsiper Sep 9, 2025
3296bdd
in_dummy: validate allocations and handle exceptions
edsiper Sep 9, 2025
4e998ca
config_format: yaml: guard for env content
edsiper Sep 9, 2025
12a0576
in_dummy: fix potential leak on resolved msg
edsiper Sep 9, 2025
e886e3b
input: fix environment var translation
edsiper Sep 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions include/fluent-bit/config_format/flb_cf.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ struct flb_cf {
/* global service */
struct flb_cf_section *service;

/* config environment variables (env section, availble on YAML) */
struct mk_list env;
/* config environment variables (env section, available on YAML) */
struct mk_list env; /* list of struct flb_cf_env_var */

/* meta commands (used by fluentbit classic mode) */
struct mk_list metas;
Expand Down Expand Up @@ -141,9 +141,22 @@ void flb_cf_destroy(struct flb_cf *cf);
int flb_cf_set_origin_format(struct flb_cf *cf, int format);
void flb_cf_dump(struct flb_cf *cf);

struct flb_kv *flb_cf_env_property_add(struct flb_cf *cf,
char *k_buf, size_t k_len,
char *v_buf, size_t v_len);
struct flb_cf_env_var {
flb_sds_t name;
flb_sds_t value;
flb_sds_t uri;
int refresh_interval;
struct mk_list _head;
};

struct flb_cf_env_var *flb_cf_env_var_add(struct flb_cf *cf,
char *name, size_t name_len,
char *value, size_t value_len,
char *uri, size_t uri_len,
int refresh_interval);

#define flb_cf_env_property_add(cf, name, name_len, value, value_len) \
flb_cf_env_var_add(cf, name, name_len, value, value_len, NULL, 0, 0)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

❓ Verification inconclusive

Compatibility macro changes return type — likely API break.
Old flb_cf_env_property_add returned struct flb_kv*; the new macro returns struct flb_cf_env_var*. Call sites assigning to struct flb_kv* will fail.

Prefer a deprecated wrapper preserving the old signature to keep source/API compatibility:

-#define flb_cf_env_property_add(cf, name, name_len, value, value_len) \
-    flb_cf_env_var_add(cf, name, name_len, value, value_len, NULL, 0, 0)
+/* Deprecated: kept for source/API compatibility. Returns NULL. */
+static inline struct flb_kv *flb_cf_env_property_add(struct flb_cf *cf,
+                                                     char *name, size_t name_len,
+                                                     char *value, size_t value_len)
+{
+    (void) flb_cf_env_var_add(cf, name, name_len, value, value_len, NULL, 0, 0);
+    return NULL;
+}

If third parties may rely on the old exported symbol, also add a compat function in the .c to preserve ABI.

Preserve flb_cf_env_property_add API compatibility
flb_cf_env_property_add was changed from returning struct flb_kv* to invoking flb_cf_env_var_add (returning struct flb_cf_env_var*), which breaks third-party code expecting a flb_kv*. Supply a deprecated inline shim with the original signature:

-#define flb_cf_env_property_add(cf, name, name_len, value, value_len) \
-    flb_cf_env_var_add(cf, name, name_len, value, value_len, NULL, 0, 0)
+/* Deprecated: kept for source/API compatibility (returns NULL). */
+static inline struct flb_kv *flb_cf_env_property_add(struct flb_cf *cf,
+                                                     char *name, size_t name_len,
+                                                     char *value, size_t value_len)
+{
+    (void) flb_cf_env_var_add(cf, name, name_len, value, value_len, NULL, 0, 0);
+    return NULL;
+}

If ABI compatibility is required, also define and export a compat symbol in the .c implementation.


/* metas */
struct flb_kv *flb_cf_meta_property_add(struct flb_cf *cf, char *meta, int len);
Expand Down
9 changes: 8 additions & 1 deletion include/fluent-bit/flb_config_map.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
#define FLB_CONFIG_MAP_VARIANT 50 /* variant that wraps a kvlist or array */

#define FLB_CONFIG_MAP_MULT 1
#define FLB_CONFIG_MAP_DYNAMIC_ENV 2 /* flag: resolve environment variables at runtime */

struct flb_config_map_val {
union {
Expand Down Expand Up @@ -108,6 +109,12 @@ struct mk_list *flb_config_map_create(struct flb_config *config,
struct flb_config_map *map);
void flb_config_map_destroy(struct mk_list *list);
int flb_config_map_expected_values(int type);
int flb_config_map_set(struct mk_list *properties, struct mk_list *map, void *context);
int flb_config_map_set(struct flb_config *config, struct mk_list *properties, struct mk_list *map, void *context);

Comment on lines +112 to +113
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

API break: flb_config_map_set now requires config; verify all call sites and out-of-tree impact

This is a source/ABI-breaking change for plugins/tests calling the old 3-arg form. Ensure all in-repo uses are updated and call out the break in release notes for external plugins.

🏁 Script executed:

#!/bin/bash
# List calls to flb_config_map_set with 3 vs 4 arguments.
#  - Requires ast-grep in the runner.
echo "Calls with 3 args (old API):"
ast-grep --pattern $'flb_config_map_set($_, $_, $_)' || true
echo ""
echo "Calls with 4 args (new API):"
ast-grep --pattern $'flb_config_map_set($_, $_, $_, $_)' || true

Length of output: 58

Document API signature change for flb_config_map_set No in-repo calls remain using the old 3-arg form; update the project’s release notes to highlight this source/ABI break for external plugins.

/* Helper function to check if a config map entry has dynamic environment resolution */
static inline int flb_config_map_has_dynamic_env(struct flb_config_map *map)
{
return (map->flags & FLB_CONFIG_MAP_DYNAMIC_ENV) != 0;
}

#endif
2 changes: 1 addition & 1 deletion include/fluent-bit/flb_custom.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ struct flb_custom_instance {
static inline int flb_custom_config_map_set(struct flb_custom_instance *ins,
void *context)
{
return flb_config_map_set(&ins->properties, ins->config_map, context);
return flb_config_map_set(ins->config, &ins->properties, ins->config_map, context);
}

int flb_custom_set_property(struct flb_custom_instance *ins,
Expand Down
14 changes: 14 additions & 0 deletions include/fluent-bit/flb_env.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,24 @@

#include <fluent-bit/flb_info.h>
#include <fluent-bit/flb_sds.h>
#include <monkey/mk_core.h>
#include <time.h>

#define FLB_ENV_SIZE 64

struct flb_env_var {
flb_sds_t name; /* variable name */
flb_sds_t value; /* value (resolved) */
flb_sds_t uri; /* optional file URI */
int refresh_interval; /* refresh interval in seconds */
time_t last_refresh; /* last refresh timestamp */
struct mk_list _head; /* link to flb_env vars list */
};

struct flb_env {
int warn_unused; /* warn about unused environment variable */
struct flb_hash_table *ht;
struct mk_list vars; /* list of environment variables */
};

static inline void flb_env_warn_unused(struct flb_env *env, int warn)
Expand All @@ -38,6 +50,8 @@ static inline void flb_env_warn_unused(struct flb_env *env, int warn)
struct flb_env *flb_env_create();
void flb_env_destroy(struct flb_env *env);
int flb_env_set(struct flb_env *env, const char *key, const char *val);
int flb_env_set_extended(struct flb_env *env, const char *key, const char *val,
const char *uri, int refresh_interval);
const char *flb_env_get(struct flb_env *env, const char *key);
flb_sds_t flb_env_var_translate(struct flb_env *env, const char *value);

Expand Down
2 changes: 1 addition & 1 deletion include/fluent-bit/flb_filter.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ struct mk_list *flb_filter_get_global_config_map(struct flb_config *config);
static inline int flb_filter_config_map_set(struct flb_filter_instance *ins,
void *context)
{
return flb_config_map_set(&ins->properties, ins->config_map, context);
return flb_config_map_set(ins->config, &ins->properties, ins->config_map, context);
}

int flb_filter_set_property(struct flb_filter_instance *ins,
Expand Down
4 changes: 2 additions & 2 deletions include/fluent-bit/flb_input.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -721,7 +721,7 @@ static inline int flb_input_config_map_set(struct flb_input_instance *ins,

/* Process normal properties */
if (ins->config_map) {
ret = flb_config_map_set(&ins->properties, ins->config_map, context);
ret = flb_config_map_set(ins->config, &ins->properties, ins->config_map, context);

if (ret == -1) {
return -1;
Expand All @@ -730,7 +730,7 @@ static inline int flb_input_config_map_set(struct flb_input_instance *ins,

/* Net properties */
if (ins->net_config_map) {
ret = flb_config_map_set(&ins->net_properties, ins->net_config_map,
ret = flb_config_map_set(ins->config, &ins->net_properties, ins->net_config_map,
&ins->net_setup);
if (ret == -1) {
return -1;
Expand Down
4 changes: 2 additions & 2 deletions include/fluent-bit/flb_output.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1302,15 +1302,15 @@ static inline int flb_output_config_map_set(struct flb_output_instance *ins,

/* Process normal properties */
if (ins->config_map) {
ret = flb_config_map_set(&ins->properties, ins->config_map, context);
ret = flb_config_map_set(ins->config, &ins->properties, ins->config_map, context);
if (ret == -1) {
return -1;
}
}

/* Net properties */
if (ins->net_config_map) {
ret = flb_config_map_set(&ins->net_properties, ins->net_config_map,
ret = flb_config_map_set(ins->config, &ins->net_properties, ins->net_config_map,
&ins->net_setup);
if (ret == -1) {
return -1;
Expand Down
2 changes: 1 addition & 1 deletion include/fluent-bit/flb_processor.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ static inline int flb_processor_instance_config_map_set(
struct flb_processor_instance *ins,
void *context)
{
return flb_config_map_set(&ins->properties, ins->config_map, context);
return flb_config_map_set(ins->config, &ins->properties, ins->config_map, context);
}

static inline
Expand Down
Loading
Loading