Skip to content

Commit b490e1f

Browse files
committed
add rule based auto tagging user guide PR
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
1 parent 4a362d5 commit b490e1f

File tree

1 file changed

+182
-0
lines changed
  • _tuning-your-cluster/availability-and-recovery/workload-management

1 file changed

+182
-0
lines changed
Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
---
2+
layout: default
3+
title: Auto-tagging Feature Overview
4+
nav_order: 20
5+
parent: Workload management
6+
grand_parent: Availability and recovery
7+
---
8+
9+
# Rule-based Auto-tagging in OpenSearch
10+
11+
## What is Rule-based Auto-tagging?
12+
13+
Rule-based Auto-tagging automatically assigns workload groups to incoming search requests in OpenSearch. This feature helps you implement automated workload management policies without manual intervention.
14+
15+
## Key Concepts
16+
17+
- **Rule**: Defines criteria for tagging search requests
18+
- **Index Pattern**: Prefix-based pattern matching target indices (e.g., "logs-*")
19+
- **Workload Group**: Label assigned to requests for workload management
20+
- **Auto-tagging**: Process of assigning workload groups based on rules
21+
- **Pattern Specificity**: More specific patterns take precedence e,g; `logs-prod-2025` will match `logs-prod-*` pattern over `logs-*`
22+
23+
## How to Set Up Rule-based Auto-tagging
24+
25+
### Before You Begin
26+
27+
- Ensure you have an OpenSearch cluster with the workload-management plugin installed
28+
- Verify you have administrative access to the cluster
29+
30+
### Managing Rules
31+
32+
Create or update a rule:
33+
```http
34+
PUT /_rules/workload_group
35+
{
36+
"description": "Production Logs Rule",
37+
"index_pattern": ["prod-logs-*"],
38+
"workload_group": "production_workload"
39+
}
40+
```
41+
42+
List all rules:
43+
```http
44+
GET /_rules/workload_group
45+
```
46+
47+
Delete a rule:
48+
```http
49+
DELETE /_rules/workload_group/{rule_id}
50+
```
51+
52+
### Rule Structure
53+
```json
54+
{
55+
"_id": "fwehf8302582mglfio349==", // System-generated
56+
"description": "Assign Query Group for Index Logs123",
57+
"index_pattern": ["logs123"], // Exact match or prefix pattern only
58+
"workload_group": "dev_workload_group_id",
59+
"updated_at": "01-10-2025T21:23:21.456Z" // System-generated timestamp
60+
}
61+
```
62+
63+
## How Pattern Matching Works
64+
65+
### Supported Pattern Types
66+
67+
1. Exact matches: `logs-2025-04`
68+
2. Prefix patterns: `logs-2025-*`
69+
70+
Note: OpenSearch doesn't support suffix patterns (`*-logs`) or generic patterns (`*-logs-*`).
71+
72+
### Pattern Precedence
73+
74+
1. Exact matches have highest priority
75+
2. Longer prefix patterns take precedence over shorter ones
76+
Example: `logs-prod-2025-*` is more specific than `logs-prod-*`
77+
78+
### Evaluation Process
79+
80+
1. OpenSearch receives a search request
81+
2. The system compares request indices against defined rules
82+
3. The most specific matching rule's workload group is applied
83+
4. If no rules match, no workload group is assigned
84+
85+
## Examples
86+
87+
### Production vs Development Logs
88+
89+
```json
90+
// Rule 1: Production Logs
91+
{
92+
"description": "Production Logs",
93+
"index_pattern": ["logs-prod-*"],
94+
"workload_group": "production"
95+
}
96+
97+
// Rule 2: Development Logs
98+
{
99+
"description": "Development Logs",
100+
"index_pattern": ["logs-dev-*"],
101+
"workload_group": "development"
102+
}
103+
104+
// Example: Production search
105+
GET /logs-prod-2025/_search
106+
// Result: Tagged with "production"
107+
108+
// Example: Development search
109+
GET /logs-dev-2025/_search
110+
// Result: Tagged with "development"
111+
```
112+
113+
### Handling Specificity
114+
115+
```json
116+
// Rule 1: General Logs
117+
{
118+
"description": "General Logs",
119+
"index_pattern": ["logs-*"],
120+
"workload_group": "general"
121+
}
122+
123+
// Rule 2: Production Service Logs
124+
{
125+
"description": "Production Service Logs",
126+
"index_pattern": ["logs-prod-service-*"],
127+
"workload_group": "prod_service"
128+
}
129+
130+
// Example: Specific production service search
131+
GET /logs-prod-service-2025/_search
132+
// Result: Tagged with "prod_service" (more specific match wins)
133+
```
134+
135+
## Benefits of Rule-based Auto-tagging
136+
137+
- Automates request tagging
138+
- Ensures consistent policy application
139+
- Scales to new indices automatically
140+
- Reduces administrative overhead
141+
- Minimizes manual errors
142+
- Centralizes workload management
143+
- Allows easy policy updates
144+
145+
## Best Practices
146+
147+
### Designing Rules
148+
149+
1. Use specific prefix patterns for precise control
150+
2. Clearly document each rule's purpose
151+
3. Use consistent workload group naming
152+
4. Create a hierarchical pattern structure
153+
154+
### Creating Patterns
155+
156+
1. Start with the most specific patterns needed
157+
2. Use consistent delimiters in index names
158+
3. Avoid unintended pattern overlaps
159+
4. Plan for future index naming conventions
160+
161+
### Operations
162+
163+
1. Test new rules in a development environment
164+
2. Monitor rule matches in your logs
165+
3. Keep rule configurations documented
166+
4. Regularly review rule effectiveness
167+
168+
## Troubleshooting
169+
170+
**Common issues and solutions:**
171+
172+
1. **No Workload Group Assigned**: Ensure your index pattern is a valid prefix
173+
2. **Unexpected Workload Group**: Look for more specific matching patterns
174+
3. **Rule Not Working**: Verify the pattern follows the prefix-only format
175+
176+
**To validate your setup:**
177+
178+
179+
- Test new rules with sample requests before production use
180+
- Use the list rules API to verify pattern matching
181+
- Check your logs for rule evaluation results
182+

0 commit comments

Comments
 (0)