Cargo test failed unless using filter

[doki23]

I ran cargo test -v in my mac m1 and got the result

failures:

---- meta_node::meta_node_all::test_meta_node_snapshot_replication stdout ----
Error: timeout after 10s when non-voter replicated all logs .last_log_index -> Some(15) latest: Metrics{id:1,Learner, term:1, last_log:None, last_applied:None, leader:Some(0), membership:{log_id:0-0 membership:[{1}]}, snapshot:None, replication:
thread 'meta_node::meta_node_all::test_meta_node_snapshot_replication' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/76d770ac21d9521db6a92a48c7b3d5b2cc535941/library/test/src/lib.rs:186:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- grpc::metasrv_grpc_kv_api_restart_cluster::test_kv_api_restart_cluster_write_read stdout ----
Error: timeout after 30s when a leader is observed latest: Metrics{id:0,Follower, term:1, last_log:Some(11), last_applied:Some(LogId { term: 1, index: 11 }), leader:None, membership:{log_id:1-8 membership:[{0, 1, 2}]}, snapshot:None, replication:
thread 'grpc::metasrv_grpc_kv_api_restart_cluster::test_kv_api_restart_cluster_write_read' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/76d770ac21d9521db6a92a48c7b3d5b2cc535941/library/test/src/lib.rs:186:5

---- meta_node::meta_node_all::test_meta_node_add_database stdout ----
Error: timeout after 10s when  .last_applied -> Some(7) latest: Metrics{id:1,Follower, term:1, last_log:Some(7), last_applied:Some(LogId { term: 1, index: 6 }), leader:Some(0), membership:{log_id:1-7 membership:[{0, 1, 2}]}, snapshot:None, replication:
thread 'meta_node::meta_node_all::test_meta_node_add_database' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/76d770ac21d9521db6a92a48c7b3d5b2cc535941/library/test/src/lib.rs:186:5


failures:
    grpc::metasrv_grpc_kv_api_restart_cluster::test_kv_api_restart_cluster_write_read
    meta_node::meta_node_all::test_meta_node_add_database
    meta_node::meta_node_all::test_meta_node_snapshot_replication

I doubt there may be some parallel problems.
And then, I reran cargo test -j 1 -- --test-threads=1, the number of failures reduces to 1:

failures:

---- meta_node::meta_node_all::test_meta_node_snapshot_replication stdout ----
Error: timeout after 10s when non-voter replicated all logs .last_log_index -> Some(15) latest: Metrics{id:1,Learner, term:1, last_log:None, last_applied:None, leader:Some(0), membership:{log_id:0-0 membership:[{1}]}, snapshot:None, replication:
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/76d770ac21d9521db6a92a48c7b3d5b2cc535941/library/test/src/lib.rs:186:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    meta_node::meta_node_all::test_meta_node_snapshot_replication

It works but not quite.

And I tried with a filter cargo test test_meta_node_snapshot_replication, it succeed.
Need help!

[Xuanwo]

Call @drmingdrmer and @lichuang for a hand. Can you take a look of this issue?

[drmingdrmer]

Yes, normally running tests with too many threads, the threads will block each other...

[Xuanwo]

What we can do to mitigate these problems?

[drmingdrmer]

• 🤔 working on it : Two frequently failed case #4864