Skip to content

Control Plane not exiting on panic #5787

@UN0wen

Description

@UN0wen

Describe the bug
We're deploying quickwit in cluster mode through helm (chart version 7.15, quickwit version 0.8.2). We noticed today that some new pods are spinning up but not connected to the existing cluster, instead only peering between themselves and crashing since there are no metastore.

Upon inspection, we found this traceback in the control plane logs (the pod is healthy)

thread 'tokio-runtime-worker' panicked at /usr/local/cargo/git/checkouts/chitchat-22cf90d3696646d6/d039699/chitchat/src/delta.rs:413:9:
assertion failed: mtu >= 100
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It seems like after encountering this error, the control plane does not exit and continues to run, seemingly doing nothing (?).

Expected behavior
Control plane should exit with an error if an internal component panics.

Configuration:
Please provide:

  1. Output of quickwit --version

quickwit version: 0.8.2 (aarch64-unknown-linux-gnu 2024-06-17T16:36:47Z 42766b8)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions