You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -7,41 +7,53 @@ grand_parent: Availability and recovery
7
7
---
8
8
9
9
# Rule-based auto-tagging in OpenSearch
10
-
In this guide we will walk you through how to use the rule based auto-tagging. Auto-tagging enriches the request with the target label which a consuming feature (such as workload-management) consumes.
10
+
11
+
Rule-based auto-tagging is a versatile feature in OpenSearch that automatically assigns specific values to incoming requests based on predefined rules. Each OpenSearch feature can define its own attributes for matching and values to be assigned.
12
+
11
13
## What is rule-based auto-tagging?
12
14
13
-
Rule-based Auto-tagging automatically assigns workload groups to incoming search requests in OpenSearch. This feature helps you implement automated feature policies without manual intervention.
14
-
throughout this guide I will take a sample feature which uses rule framework for auto-tagging i,e; workload-management
15
+
Rule-based auto-tagging automatically evaluates incoming requests against a set of predefined rules by matching request attributes. When a rule matches, it assigns a feature-specific value to the request. For example, the workload management feature uses index patterns as an attribute and assigns workload group IDs.
15
16
16
17
## Key concepts
17
18
18
-
-**Rule**: Defines criteria for tagging search requests
19
-
-**Attributes**: Prefix-based pattern matching for the given attribute value (e.g., attribute could be index pattern, security context values such as username, role, group etc.)
20
-
-**Label**: Label assigned to requests for the client feature (e,g; workload management feature will use 'workload_group' as label)
21
-
-**Auto-tagging**: Process of assigning workload groups based on rules
22
-
-**Pattern Specificity**: More specific patterns take precedence e,g; `logs-prod-2025` will match `logs-prod-*` pattern over `logs-*`
19
+
Before diving into the details, let's understand the key components:
23
20
24
-
## How to set up rule-based auto-tagging
25
-
Now we will learn about how can we set up the auto-tagging in opensearch.
21
+
-**Rule**: Defines matching criteria (attributes) and the value to be assigned
22
+
-**Attributes**: Key-value pairs used for matching rules (e.g., index patterns, user roles, request types)
23
+
-**Feature-specific Value**: The value to be assigned when a rule matches
24
+
-**Pattern Matching**: How attribute values are matched (exact or pattern-based)
26
25
27
-
### Before you begin
26
+
##Rule structure and management
28
27
29
-
- Ensure you have an OpenSearch cluster with the feature (e,g; workload-management uses this framework for autotagging) plugin installed
30
-
- Verify you have administrative access to the cluster
The attribute matching system determines which rules apply to a given request. Different attributes can have different matching behaviors.
66
69
67
-
### Supported pattern types
70
+
### Attribute matching types
68
71
69
-
1. Exact matches: `logs-2025-04`
70
-
2. Prefix patterns: `logs-2025-*`
72
+
Attributes can support various matching types depending on their nature:
71
73
72
-
Note: This feature doesn't support suffix patterns (`*-logs`) or generic patterns (`*-logs-*`).
74
+
1. Exact matching: Values must match exactly
75
+
2. Pattern matching: Values can match patterns (e.g., index patterns in WLM)
76
+
3. List matching: Values can match any item in a list
77
+
4. Range matching: Values can fall within defined ranges
73
78
74
-
### Pattern precedence
79
+
For example, in workload management, index patterns support:
80
+
- Exact matches: `logs-2025-04`
81
+
- Prefix patterns: `logs-2025-*`
75
82
76
-
1. Exact matches have highest priority
77
-
2. Longer prefix patterns take precedence over shorter ones
78
-
Example: `logs-prod-2025-*` is more specific than `logs-prod-*`
83
+
Note: The specific matching behavior depends on the attribute type and the feature using it.
84
+
85
+
### Rule precedence
86
+
87
+
When multiple rules match a request, precedence is determined by:
88
+
89
+
1. More specific attribute matches take priority
90
+
2. Feature-specific tie-breaking rules are applied
91
+
92
+
For example, in workload management with index patterns:
93
+
-`logs-prod-2025-*` is more specific than `logs-prod-*`
94
+
-`logs-prod-*` is more specific than `logs-*`
79
95
80
96
### Evaluation process
81
97
82
-
1. OpenSearch receives a search request
83
-
2. The system compares request indices against defined rules
84
-
3. The most specific matching rule's workload group is applied
85
-
4. If no rules match, no workload group is assigned
98
+
Here's how OpenSearch evaluates incoming requests:
99
+
100
+
1. OpenSearch receives a request
101
+
2. The system evaluates request attributes against defined rules
102
+
3. The most specific matching rule's feature value is assigned
103
+
4. If no rules match, no value is assigned
86
104
87
105
## Examples
88
106
89
-
### Production vs development logs
107
+
Let's look at some examples using workload management, which uses index patterns as its primary attribute:
108
+
109
+
### Multiple attribute matching
90
110
91
111
```json
92
-
// Rule 1: Production Logs
112
+
// Rule with multiple attributes
93
113
{
94
-
"description": "Production Logs",
95
-
"index_pattern": ["logs-prod-*"],
96
-
"workload_group": "production"
114
+
"index_patterns": ["logs-prod-*"],
115
+
"request_type": ["search", "count"], // it is used here just for demonstration purposes since WLM only supports index pattern at present
116
+
"workload_group": "production_search_workload_id"
97
117
}
98
118
99
-
// Rule 2: Development Logs
119
+
// Rule with single attribute
100
120
{
101
-
"description": "Development Logs",
102
-
"index_pattern": ["logs-dev-*"],
103
-
"workload_group": "development"
121
+
"index_patterns": ["logs-prod-*"],
122
+
"workload_group": "production_workload_id"
104
123
}
105
-
106
-
// Example: Production search
107
-
GET /logs-prod-2025/_search
108
-
// Result: Tagged with "production"
109
-
110
-
// Example: Development search
111
-
GET /logs-dev-2025/_search
112
-
// Result: Tagged with "development"
113
124
```
114
125
115
-
### Handling specificity
126
+
### Attribute specificity
116
127
117
128
```json
118
-
// Rule 1: General Logs
129
+
// Rule 1: General matching
119
130
{
120
-
"description": "General Logs",
121
-
"index_pattern": ["logs-*"],
122
-
"workload_group": "general"
131
+
"index_patterns": ["logs-*"],
132
+
"workload_group": "general_workload_id"
123
133
}
124
134
125
-
// Rule 2: Production Service Logs
135
+
// Rule 2: More specific matching
126
136
{
127
-
"description": "Production Service Logs",
128
-
"index_pattern": ["logs-prod-service-*"],
129
-
"workload_group": "prod_service"
137
+
"index_patterns": ["logs-prod-service-*"],
138
+
"workload_group": "prod_service_workload_id"
130
139
}
131
-
132
-
// Example: Specific production service search
133
-
GET /logs-prod-service-2025/_search
134
-
// Result: Tagged with "prod_service" (more specific match wins)
135
140
```
136
141
137
142
## Benefits of rule-based auto-tagging
138
143
139
-
- Automates request tagging
140
-
- Ensures consistent policy application
141
-
- Scales to new attributes automatically
142
-
- Reduces administrative overhead
143
-
- Minimizes manual errors
144
-
- Allows easy policy updates
144
+
Rule-based auto-tagging offers several advantages:
145
+
146
+
- Flexible attribute-based matching
147
+
- Support for feature-specific matching logic
148
+
- Consistent policy application
149
+
- Automated request classification
150
+
- Reduced administrative overhead
151
+
- Centralized rule management
152
+
- Easy policy updates
145
153
146
154
## Best practices
147
-
The following best practices will help you familiarize with correct usage and avoid common pitfalls
155
+
156
+
To get the most out of rule-based auto-tagging, consider these best practices:
148
157
149
158
### Designing rules
150
159
151
-
1. Use specific prefix patterns for precise control
152
-
2. Clearly document each rule's purpose
153
-
3. Create a hierarchical pattern structure
160
+
1. Identify the most relevant attributes for your use case
161
+
2. Use specific attribute values for precise control
162
+
3. Combine multiple attributes when needed
163
+
4. Use consistent naming conventions
164
+
5. Document attribute matching behavior
154
165
155
-
### Creating patterns
166
+
### Managing attributes
156
167
157
-
1.Start with the most specific patterns needed
158
-
2.Use consistent delimiters in index names
159
-
3. Avoid unintended pattern overlaps
160
-
4. Plan for future index naming conventions
168
+
1.Understand each attribute's matching behavior
169
+
2.Start with the most specific criteria needed
170
+
3. Avoid overlapping rules unless intentional
171
+
4. Plan for future attribute value patterns
161
172
162
173
### Operations
163
174
164
175
1. Test new rules in a development environment
165
176
2. Monitor rule matches in your logs
166
-
3.Keep rule configurations documented
177
+
3.Document rule configurations
167
178
4. Regularly review rule effectiveness
179
+
5. Clean up unused rules
168
180
169
181
## Troubleshooting
170
182
171
-
**Common issues and solutions:**
183
+
Common issues and their solutions:
184
+
185
+
1.**No value assigned**: Verify all attribute values are correct
186
+
2.**Unexpected value**: Check for more specific matching rules
187
+
3.**Rule not working**: Confirm attribute matching behavior
172
188
173
-
1.**No Label Assigned**: Ensure your index pattern is a valid prefix
174
-
2.**Unexpected Feature Label**: Look for more specific matching patterns
175
-
3.**Rule Not Working**: Verify the pattern follows the prefix-only format
189
+
To validate your setup:
176
190
177
-
**To validate your setup:**
191
+
- Test rules with sample requests
192
+
- Use the list rules API to verify configurations
193
+
- Monitor rule evaluation in logs
178
194
195
+
## Learn more
179
196
180
-
- Test new rules with sample requests before production use
181
-
- Use the list rules API to verify pattern matching
182
-
- Check the consuming feature specific logs/stats to verify the correctness
197
+
Rule-based auto-tagging provides a flexible framework for implementing feature-specific request handling. While we've used workload management as an example, the attribute-based matching system can be adapted for various use cases.
0 commit comments