Stall investigations on my local #435
Description
By running 4 nodes, with k = 2, benchmark worker 1, the consensus stalled on my local.
The last finalized block is different between the first 3 nodes and the last node.
The first 3 nodes finalized block 99677603a9d9b6d05233dea515094a10c8a39fb0aaf646b1ba6676f900c370c1,
and the last node finalized block f59467a057bc3fdaf48d9a67fbe2324a1facc29721441564f467c3768218f9db.
Then, the last proposed block by each node is different.
Node 1:
2:52:17PM INF Proposing block... block_id: c4c10a2165ef8d3cb63111a350cdd77c799cd3555f49a4c54d7c402043951d00 block_index: 13 num_transactions: 7939
Node 2:
2:52:18PM INF Proposing block... block_id: 697143118e77983b4934ca822a893b36e3d1fd46897982a4c06acbb6e6a12af7 block_index: 13 num_transactions: 5000
Node 3:
2:52:19PM INF Proposing block... block_id: 77bcd3e31b384cfcb8bc7c42d1c4639c6766128e5c3cd3e2b911d35d12245052 block_index: 13 num_transactions: 7357
Node 4:
2:52:56PM INF Proposing block... block_id: e51233c68904f88ccd6a46707e279b25357cd20ef5c3eae950107d815683970a block_index: 13 num_transactions: 28684
Commands used to run the 4 nodes.
go run -tags=debug . -port 3000 -api.port 9000 -wallet config/wallet.txt -db /private/tmp/wavelet/node1 -loglevel debug -sys.snowball.k 2 |& tee ~/Desktop/master-stall/node1.txt
go run -tags=debug . -port 3001 -api.port 9001 -wallet config/wallet2.txt -db /private/tmp/wavelet/node2 -loglevel debug -sys.snowball.k 2 127.0.0.1:3000 |& tee ~/Desktop/master-stall/node2.txt 127.0.0.1:3000
go run -tags=debug . -port 3002 -api.port 9002 -wallet config/wallet3.txt -db /private/tmp/wavelet/node3 -loglevel debug -sys.snowball.k 2 127.0.0.1:3000 |& tee ~/Desktop/master-stall/node3.txt 127.0.0.1:3000
go run -tags=debug . -port 3003 -api.port 9003 -db /private/tmp/wavelet/node4 -loglevel debug -sys.snowball.k 2 127.0.0.1:3000 |& tee ~/Desktop/master-stall/node4.txt
Below is the CLI dumps on all the nodes.
node1.txt
node2.txt
node3.txt
node4.txt