Skip to content

Commit 3830e27

Browse files
Extend the topic description for the Kafka output (#3048)
This PR extends the **Topics settings** section of the **Kafka output** docs for both standalone and Fleet-managed Elastic Agents, and: - adds missing information about setting dynamic topics in the Kafka output doc for standalone Elastic Agents, - adds additional information and an example for using the `add_fields` processor for setting complex, dynamic topic value in both docs, - fixes a typo and a link text. Resolves elastic/ingest-docs#1780
1 parent be9fd7b commit 3830e27

File tree

2 files changed

+41
-3
lines changed

2 files changed

+41
-3
lines changed

reference/fleet/kafka-output-settings.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,13 +131,27 @@ Use this option to set the Kafka topic for each {{agent}} event.
131131

132132
You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
133133

134-
* `data_stream_type`
134+
* `data_stream.type`
135135
* `data_stream.dataset`
136136
* `data_stream.namespace`
137137
* `@timestamp`
138138
* `event-dataset`
139139

140-
You can also set a custom field. This is useful if you’re using the [`add_fields` processor](/reference/fleet/add_fields-processor.md) as part of your {{agent}} input. Otherwise, setting a custom field is not recommended.
140+
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
141+
142+
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}.
143+
144+
For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
145+
146+
```yaml
147+
- add_fields:
148+
target: ''
149+
fields:
150+
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
151+
```
152+
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.
153+
154+
For more information, refer to [](/reference/fleet/agent-processors.md).
141155

142156

143157
### Header settings [_header_settings]

reference/fleet/kafka-output.md

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ inputs {
6969

7070
The `kafka` output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run {{agent}} with minimal configuration.
7171

72-
* [Kafka output configuration settings](#output-kafka-commonly-used-settings)
72+
* [Commonly used settings](#output-kafka-commonly-used-settings)
7373
* [Authentication settings](#output-kafka-authentication-settings)
7474
* [Memory queue settings](#output-kafka-memory-queue-settings)
7575
* [Topics settings](#output-kafka-topics-settings)
@@ -164,6 +164,30 @@ Use these options to set the Kafka topic for each {{agent}} event.
164164
`topic` $$$kafka-topic-setting$$$
165165
: The default Kafka topic used for produced events.
166166

167+
You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
168+
169+
* `data_stream.type`
170+
* `data_stream.dataset`
171+
* `data_stream.namespace`
172+
* `@timestamp`
173+
* `event-dataset`
174+
175+
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
176+
177+
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}.
178+
179+
For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
180+
181+
```yaml
182+
- add_fields:
183+
target: ''
184+
fields:
185+
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
186+
```
187+
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.
188+
189+
For more information, refer to [](/reference/fleet/agent-processors.md).
190+
167191

168192
## Partition settings [output-kafka-partition-settings]
169193

0 commit comments

Comments
 (0)