Adjust Elasticsearch Permissions to not require manage_security or superuser roles #390
Description
Is your feature request related to a problem? Please describe.
The problem we have is that we would like to point Assemblyline at a shared Elasticsearch cluster where we pay for Elasticsearch Enterprise licensing to enable Enterprise features and support. This cluster has many nodes and is capable of a lot of underlying storage and compute.
This is problematic since Assemblyline requires manage_security and superuser within a shared ES cluster environment since it provides elevated permissions across the entire cluster to create new users to have full access to everything.
We would generally prefer to leverage this Enterprise Elasticsearch cluster than to have AL spin up its own ES instance, but the way that Assemblyline is coded it in a way that it will be connected to an ES instance that is on-cluster and not shared with other systems given the permissions Assemblyline currently requires for provisioning the plumber user
Describe the solution you'd like
It seems possible that there could be a way to remove the need for Assemblyline to always require the manage_security and superuser role.
Looking at the following code:
This is where Assemblyline appears to always attempt to put the manage_tasks security role and put the plumber user into ES with manage_tasks and superuser
Ideally, it seems that this code could likely be tweaked in some way to remove the manage_security permission from the Assembline ES user, such that this code could attempt to connect to ES with the plumber user and if this fails then it can error indicating that the user does not exist, or if it does connect it can then check its permissions so that it doesnt need to create its user and roles each time and not exactly sure why the superuser is required since it should be possible to grant specific permissions to limit the user to limit what it can do.
Setting up the necessary user / permissions could be part of the bootstrapping process so that it only occurs one time and then when Assemblyline starts normally it can try to connect as the normal user and see if it has the proper permissions (in the event anything may change in the future) and if it is missing anything then the admin could re-initialize providing elevated access for Assemlyline to make those changes as a one time event or it could be possible to just make those permission changes completely out of band from Assemblyline.
Describe alternatives you've considered
Open to alternative suggestions, but just looking for a way to be able to restrict the Assemblyline Elasticsearch permissions at normal runtime. Open to providing elevated permissions as a one off bootstrapping / initialization process, but could theoretically manage the permissions completely out of band to ensure Assemblyline is only able to manage its own data.
** Additional Context **
Linking to another somewhat related ticket where it would be nice to support prefixing Elasticsearch indexes (#391) , which then makes it easier to manage user permissions to be scoped to Assemblyline specific indexes based on the prefix