Avoid stack overflow in IndicesClusterStateService applyClusterState

[albertzaharovits]

Every cluster state applied in the IndicesClusterStateService has the potential to chain a new RefCountingListener to a chain of such listeners. If the chain is too long, the unlucky thread that decreases the ref count to 0 for the head of the listeners chain, ends up calling each listener in turn, and, assuming all ref counts are hence decreased to 0, traversing the whole chain on its thread stack, possibly resulting in a Stackoverflow exception.

This fix chains max 8 RefCountingListener, the 11th one is forked on a generic thread when it gets to execution.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[elasticsearchmachine]

Hi @albertzaharovits, I've created a changelog YAML for you.

[albertzaharovits]

Honestly, I think I prefer that every chained listener be executed on a generic thread, for code simplicity's sake.

[DaveCTurner]

I'd rather we didn't extend the chain in the (overwhelmingly common) case where the cluster state update doesn't close any more shards.

Also can you cover this in a test?

[DaveCTurner]

Hmm are you sure we should move all this listener stuff below doApplyClusterState()?

[albertzaharovits]

I can't think of any impact to execution.

[albertzaharovits]

But I've put it back at the original place.

[albertzaharovits]

I'd rather we didn't extend the chain in the (overwhelmingly common) case where the cluster state update doesn't close any more shards.

Pushed 3a00599

[albertzaharovits]

@DaveCTurner can you take another look please?

I've changed the code to avoid linking listeners when the applied cluster state doesn't close any shards.
I've also added a test that asserts that all the runnables before the oldest shard close listener that's not complete are run, while the others are not.