You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Extend the topic description for the Kafka output (#3048)
This PR extends the **Topics settings** section of the **Kafka output**
docs for both standalone and Fleet-managed Elastic Agents, and:
- adds missing information about setting dynamic topics in the Kafka
output doc for standalone Elastic Agents,
- adds additional information and an example for using the `add_fields`
processor for setting complex, dynamic topic value in both docs,
- fixes a typo and a link text.
Resolveselastic/ingest-docs#1780
Copy file name to clipboardExpand all lines: reference/fleet/kafka-output-settings.md
+16-2Lines changed: 16 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,13 +131,27 @@ Use this option to set the Kafka topic for each {{agent}} event.
131
131
132
132
You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
133
133
134
-
* `data_stream_type`
134
+
* `data_stream.type`
135
135
* `data_stream.dataset`
136
136
* `data_stream.namespace`
137
137
* `@timestamp`
138
138
* `event-dataset`
139
139
140
-
You can also set a custom field. This is useful if you’re using the [`add_fields` processor](/reference/fleet/add_fields-processor.md) as part of your {{agent}} input. Otherwise, setting a custom field is not recommended.
140
+
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
141
+
142
+
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}.
143
+
144
+
For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.
153
+
154
+
For more information, refer to [](/reference/fleet/agent-processors.md).
Copy file name to clipboardExpand all lines: reference/fleet/kafka-output.md
+25-1Lines changed: 25 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -69,7 +69,7 @@ inputs {
69
69
70
70
The `kafka` output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run {{agent}} with minimal configuration.
@@ -164,6 +164,30 @@ Use these options to set the Kafka topic for each {{agent}} event.
164
164
`topic`$$$kafka-topic-setting$$$
165
165
: The default Kafka topic used for produced events.
166
166
167
+
You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
168
+
169
+
* `data_stream.type`
170
+
* `data_stream.dataset`
171
+
* `data_stream.namespace`
172
+
* `@timestamp`
173
+
* `event-dataset`
174
+
175
+
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
176
+
177
+
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}.
178
+
179
+
For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.
188
+
189
+
For more information, refer to [](/reference/fleet/agent-processors.md).
0 commit comments