|
| 1 | +--- |
| 2 | +layout: default |
| 3 | +title: Auto-tagging Feature Overview |
| 4 | +nav_order: 20 |
| 5 | +parent: Workload management |
| 6 | +grand_parent: Availability and recovery |
| 7 | +--- |
| 8 | + |
| 9 | +# Rule-based Auto-tagging in OpenSearch |
| 10 | + |
| 11 | +## What is Rule-based Auto-tagging? |
| 12 | + |
| 13 | +Rule-based Auto-tagging automatically assigns workload groups to incoming search requests in OpenSearch. This feature helps you implement automated workload management policies without manual intervention. |
| 14 | + |
| 15 | +## Key Concepts |
| 16 | + |
| 17 | +- **Rule**: Defines criteria for tagging search requests |
| 18 | +- **Index Pattern**: Prefix-based pattern matching target indices (e.g., "logs-*") |
| 19 | +- **Workload Group**: Label assigned to requests for workload management |
| 20 | +- **Auto-tagging**: Process of assigning workload groups based on rules |
| 21 | +- **Pattern Specificity**: More specific patterns take precedence e,g; `logs-prod-2025` will match `logs-prod-*` pattern over `logs-*` |
| 22 | + |
| 23 | +## How to Set Up Rule-based Auto-tagging |
| 24 | + |
| 25 | +### Before You Begin |
| 26 | + |
| 27 | +- Ensure you have an OpenSearch cluster with the workload-management plugin installed |
| 28 | +- Verify you have administrative access to the cluster |
| 29 | + |
| 30 | +### Managing Rules |
| 31 | + |
| 32 | +Create or update a rule: |
| 33 | +```http |
| 34 | +PUT /_rules/workload_group |
| 35 | +{ |
| 36 | + "description": "Production Logs Rule", |
| 37 | + "index_pattern": ["prod-logs-*"], |
| 38 | + "workload_group": "production_workload" |
| 39 | +} |
| 40 | +``` |
| 41 | + |
| 42 | +List all rules: |
| 43 | +```http |
| 44 | +GET /_rules/workload_group |
| 45 | +``` |
| 46 | + |
| 47 | +Delete a rule: |
| 48 | +```http |
| 49 | +DELETE /_rules/workload_group/{rule_id} |
| 50 | +``` |
| 51 | + |
| 52 | +### Rule Structure |
| 53 | +```json |
| 54 | +{ |
| 55 | + "_id": "fwehf8302582mglfio349==", // System-generated |
| 56 | + "description": "Assign Query Group for Index Logs123", |
| 57 | + "index_pattern": ["logs123"], // Exact match or prefix pattern only |
| 58 | + "workload_group": "dev_workload_group_id", |
| 59 | + "updated_at": "01-10-2025T21:23:21.456Z" // System-generated timestamp |
| 60 | +} |
| 61 | +``` |
| 62 | + |
| 63 | +## How Pattern Matching Works |
| 64 | + |
| 65 | +### Supported Pattern Types |
| 66 | + |
| 67 | +1. Exact matches: `logs-2025-04` |
| 68 | +2. Prefix patterns: `logs-2025-*` |
| 69 | + |
| 70 | +Note: OpenSearch doesn't support suffix patterns (`*-logs`) or generic patterns (`*-logs-*`). |
| 71 | + |
| 72 | +### Pattern Precedence |
| 73 | + |
| 74 | +1. Exact matches have highest priority |
| 75 | +2. Longer prefix patterns take precedence over shorter ones |
| 76 | + Example: `logs-prod-2025-*` is more specific than `logs-prod-*` |
| 77 | + |
| 78 | +### Evaluation Process |
| 79 | + |
| 80 | +1. OpenSearch receives a search request |
| 81 | +2. The system compares request indices against defined rules |
| 82 | +3. The most specific matching rule's workload group is applied |
| 83 | +4. If no rules match, no workload group is assigned |
| 84 | + |
| 85 | +## Examples |
| 86 | + |
| 87 | +### Production vs Development Logs |
| 88 | + |
| 89 | +```json |
| 90 | +// Rule 1: Production Logs |
| 91 | +{ |
| 92 | + "description": "Production Logs", |
| 93 | + "index_pattern": ["logs-prod-*"], |
| 94 | + "workload_group": "production" |
| 95 | +} |
| 96 | + |
| 97 | +// Rule 2: Development Logs |
| 98 | +{ |
| 99 | + "description": "Development Logs", |
| 100 | + "index_pattern": ["logs-dev-*"], |
| 101 | + "workload_group": "development" |
| 102 | +} |
| 103 | + |
| 104 | +// Example: Production search |
| 105 | +GET /logs-prod-2025/_search |
| 106 | +// Result: Tagged with "production" |
| 107 | + |
| 108 | +// Example: Development search |
| 109 | +GET /logs-dev-2025/_search |
| 110 | +// Result: Tagged with "development" |
| 111 | +``` |
| 112 | + |
| 113 | +### Handling Specificity |
| 114 | + |
| 115 | +```json |
| 116 | +// Rule 1: General Logs |
| 117 | +{ |
| 118 | + "description": "General Logs", |
| 119 | + "index_pattern": ["logs-*"], |
| 120 | + "workload_group": "general" |
| 121 | +} |
| 122 | + |
| 123 | +// Rule 2: Production Service Logs |
| 124 | +{ |
| 125 | + "description": "Production Service Logs", |
| 126 | + "index_pattern": ["logs-prod-service-*"], |
| 127 | + "workload_group": "prod_service" |
| 128 | +} |
| 129 | + |
| 130 | +// Example: Specific production service search |
| 131 | +GET /logs-prod-service-2025/_search |
| 132 | +// Result: Tagged with "prod_service" (more specific match wins) |
| 133 | +``` |
| 134 | + |
| 135 | +## Benefits of Rule-based Auto-tagging |
| 136 | + |
| 137 | +- Automates request tagging |
| 138 | +- Ensures consistent policy application |
| 139 | +- Scales to new indices automatically |
| 140 | +- Reduces administrative overhead |
| 141 | +- Minimizes manual errors |
| 142 | +- Centralizes workload management |
| 143 | +- Allows easy policy updates |
| 144 | + |
| 145 | +## Best Practices |
| 146 | + |
| 147 | +### Designing Rules |
| 148 | + |
| 149 | +1. Use specific prefix patterns for precise control |
| 150 | +2. Clearly document each rule's purpose |
| 151 | +3. Use consistent workload group naming |
| 152 | +4. Create a hierarchical pattern structure |
| 153 | + |
| 154 | +### Creating Patterns |
| 155 | + |
| 156 | +1. Start with the most specific patterns needed |
| 157 | +2. Use consistent delimiters in index names |
| 158 | +3. Avoid unintended pattern overlaps |
| 159 | +4. Plan for future index naming conventions |
| 160 | + |
| 161 | +### Operations |
| 162 | + |
| 163 | +1. Test new rules in a development environment |
| 164 | +2. Monitor rule matches in your logs |
| 165 | +3. Keep rule configurations documented |
| 166 | +4. Regularly review rule effectiveness |
| 167 | + |
| 168 | +## Troubleshooting |
| 169 | + |
| 170 | +**Common issues and solutions:** |
| 171 | + |
| 172 | +1. **No Workload Group Assigned**: Ensure your index pattern is a valid prefix |
| 173 | +2. **Unexpected Workload Group**: Look for more specific matching patterns |
| 174 | +3. **Rule Not Working**: Verify the pattern follows the prefix-only format |
| 175 | + |
| 176 | +**To validate your setup:** |
| 177 | + |
| 178 | + |
| 179 | +- Test new rules with sample requests before production use |
| 180 | +- Use the list rules API to verify pattern matching |
| 181 | +- Check your logs for rule evaluation results |
| 182 | + |
0 commit comments