Merge branch 'main' into log-actor-exit-phase

Author: fulmicoton-dd

.github/workflows/publish_docker_images.yml

@@ -102,7 +102,7 @@ jobs:
     environment: production
     steps:
       - name: Download digests
-        uses: actions/download-artifact@v4.2.1
+        uses: actions/download-artifact@v4.3.0
         with:
           pattern: digest-*
           path: /tmp/digests

LICENSE-3rdparty.csv

@@ -141,7 +141,7 @@ fast:
 
 | Tokenizer     | Description   |
 | ------------- | ------------- |
-| `raw`         | Does not process nor tokenize text. Filters out tokens larger than 255 bytes.  |
+| `raw`         | Does not process nor tokenize text. Filters out tokens larger than 255 bytes. This is similar to the `keyword` type in Elasticsearch. |
 | `raw_lowercase` | Does not tokenize text, but lowercase it. Filters out tokens larger than 255 bytes.  |
 | `default`     | Chops the text on according to whitespace and punctuation, removes tokens that are too long, and converts to lowercase. Filters out tokens larger than 255 bytes. |
 | `en_stem`     | Like `default`, but also applies stemming on the resulting tokens. Filters out tokens larger than 255 bytes.  |

docs/configuration/index-config.md

@@ -178,6 +178,7 @@ indexer:
 | `max_queue_memory_usage` | Maximum size in bytes of the in-memory Ingest queue. | `2GiB` |
 | `max_queue_disk_usage` | Maximum disk-space in bytes taken by the Ingest queue. The minimum size is at least `256M` and be at least `max_queue_memory_usage`. | `4GiB` |
 | `content_length_limit` | Maximum payload size uncompressed. Increasing this is discouraged, use a [file source](../ingest-data/sqs-files.md) instead. | `10MiB` |
+| `grpc_compression_algorithm` | Compression algorithm (`gzip` or `zstd`) to use for gRPC traffic between nodes for the ingest service | `None` |
 
 Example:
 
@@ -186,6 +187,7 @@ ingest_api:
   max_queue_memory_usage: 2GiB
   max_queue_disk_usage: 4GiB
   content_length_limit: 10MiB
+  grpc_compression_algorithm: zstd
 ```
 
 ## Searcher configuration

docs/configuration/node-config.md

@@ -90,7 +90,7 @@ The Google Cloud Storage flavor (`gcs`) turns off multi-object delete requests a
 
 *MinIO flavor*
 
-The MinIO flavor (`minio`) forces path-style access.
+The MinIO flavor (`minio`) overrides the `region` parameter to `minio` and forces path-style access.
 
 Example of a storage configuration for Google Cloud Storage in YAML format:
 

docs/configuration/storage-config.md

@@ -1,5 +1,5 @@
 ---
-title: Version 0.7 upgrade
+title: Version upgrade
 sidebar_position: 4
 ---
 
@@ -23,4 +23,22 @@ No migration is done if `otel-traces-v0_7` already exists. If you want `service_
 
 Quickwit 0.9 introduces a new ingestion service to to power the ingest and bulk APIs (v2). The new ingest is enabled and used by default, even though the legacy one (v1) remains enabled to finish indexing residual data in the legacy write ahead logs. Note that `ingest_api.max_queue_disk_usage` is enforced on both ingest versions separately, which means that the cumulated disk usage might be up to twice this limit.
 
-The control plane should be upgraded first in order to enable the new ingest source (v2) on all existing indexes. Ingested data into previously existing indexes on upgraded indexer nodes will not be picked by the indexing pipelines until the control plane is upgraded. Because the indexing plan is computed differently in 0.9, all pipelines will be restarted when upgrading the control plane. If possible, we recommend avoiding rolling upgrades for indexers. Instead, scale down the number of indexers to zero first, then upgrade the control plane and finally scale the upgraded indexers back up.
+When upgrading to 0.9, we recommend to perform a full cluster restart.
+
+<!--
+Reasons:
+- Ingested data into previously existing indexes on upgraded indexer nodes will not be picked by the indexing pipelines until the control plane is upgraded.
+- The indexing plan is computed differently in 0.9, all pipelines will be restarted when upgrading the control plane.
+- If you intend to enable compression for the ingest service (`ingest_api.grpc_compression_algorithm`), you must do so in two steps: first, upgrade the indexer nodes with compression disabled, then update the node configuration to enable compression, and finally restart the indexer nodes.
+- Obscure bug raised in https://github.com/quickwit-oss/quickwit/issues/5787#issuecomment-2979470315
+-->
+
+Shutdown order:
+1) indexers, searchers and janitor
+2) control plane
+3) metastores
+
+Start up order:
+1) metastores
+2) control plane
+3) indexers, searchers and janitor

docs/operating/upgrades.md

@@ -199,6 +199,7 @@ quickwit index update
 |-----------------|-------------|
 | `--index` | ID of the target index |
 | `--index-config` | Location of the index config file. |
+| `--create` | Create the index if it doesn't exist. |
 ### index clear
 
 Clears an index: deletes all splits and resets checkpoint.  
@@ -514,6 +515,7 @@ quickwit source update
 | `--index` | ID of the target index |
 | `--source` | ID of the source |
 | `--source-config` | Path to source config file. Please, refer to the documentation for more details. |
+| `--create` | Create the source if it doesn't exist. |
 ### source enable
 
 Enables a source for an index.  

docs/reference/cli.md

@@ -672,6 +672,12 @@ Moreover, while Quickwit does not support `best_fields` or `cross_fields`, it wi
 
 [Elasticsearch reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl-term-query.html)
 
+:::note
+
+When working on text, it is recommended to only use `term` queries on fields configured with `tokenizer: raw`. This is the Quickwit equivalent of the Elasticsearch `keyword` type.
+
+:::
+
 #### Example
 
 ```json

docs/reference/es_compatible_api.md

@@ -327,7 +327,19 @@ The response is the index metadata of the created index, and the content type is
 PUT api/v1/indexes/<index id>
 ```
 
-Updates the configurations of an index. This endpoint follows PUT semantics, which means that all the fields of the current configuration are replaced by the values specified in this request or the associated defaults. In particular, if the field is optional (e.g. `retention_policy`), omitting it will delete the associated configuration. If the new configuration file contains updates that cannot be applied, the request fails, and none of the updates are applied. The API accepts JSON with `content-type: application/json` and YAML with `content-type: application/yaml`.
+#### Path variable
+
+| Variable      | Description   |
+| ------------- | ------------- |
+| `index id`    | The index id  |
+
+#### Query parameters
+
+| Variable  | Type   | Description                                   | Default value |
+|-----------|--------|-----------------------------------------------|---------------|
+| `create`  | `bool` | Create the index if it doesn't already exists | `false`       |
+
+Update the configurations of an index. This endpoint follows PUT semantics, which means that all the fields of the current configuration are replaced by the values specified in this request or the associated defaults. In particular, if the field is optional (e.g. `retention_policy`), omitting it will delete the associated configuration. If the new configuration file contains updates that cannot be applied, the request fails, and none of the updates are applied. The API accepts JSON with `content-type: application/json` and YAML with `content-type: application/yaml`.
 
 - The retention policy update is automatically picked up by the janitor service on its next state refresh.
 - The search settings update is automatically picked up by searcher nodes when the next query is executed.
@@ -645,6 +657,19 @@ The response is the created source config, and the content type is `application/
 PUT api/v1/indexes/<index id>/sources/<source id>
 ```
 
+#### Path variable
+
+| Variable      | Description   |
+| ------------- | ------------- |
+| `index id`    | The index id  |
+| `source id`   | The source id  |
+
+#### Query parameters
+
+| Variable  | Type   | Description                                   | Default value |
+|-----------|--------|-----------------------------------------------|---------------|
+| `create`  | `bool` | Create the index if it doesn't already exists | `false`       |
+
 Update a source by posting a source config JSON payload.
 
 #### PUT payload

docs/reference/rest-api.md

@@ -44,7 +44,7 @@ Conversion from a type to itself is omitted. Conversions that never succeed and
 | date | text | convert to rfc3339 textual representation |
 | ip | text | convert to IPv6 representation. For IPv4, convert to IPv4-mapped IPv6 address (`::ffff:1.2.3.4`) |
 | bool | text | convert to "true" or false" |
-| u64/i64/f64 | bool | convert 0/0.0 to false and 1/1.0 to true, otherise omit |
+| u64/i64/f64 | bool | convert 0/0.0 to false and 1/1.0 to true, otherwise omit |
 | text | bool | convert if "true" or "false" (lowercase), otherwise omit |
 | text | ip | convert if valid IPv4 or IPv6, otherwise omit |
 | text | f64 | convert if valid floating point number, otherwise omit |