MPI Refactor

[wilfonba]

Description

This PR refactors a lot of MPI code to reduce duplicated code and shorten the MPI-related code in the codebase. Significant testing is needed to verify the changes' correctness, but I'm opening this as a draft now so that people know what's being changed and can start reviews/make suggestions early.

Type of change

Please delete options that are not relevant.

• Something else

Scope

• This PR comprises a set of related changes with a common goal

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

• 3D IBM one rank vs. multi-rank comparison (simple geometry)
• 2D IBM one rank vs. multi-rank comparison (simple geometry)
• 3D IBM one rank vs. multi-rank comparison (complex geometry)
• 2D IBM one rank vs. multi-rank comparison (complex geometry)
• 3D surface tension one rank vs. multi-rank comparison
• 2D surface tension one rank vs. multi-rank comparison
• 3D QBMM one rank vs. multi-rank comparison
• 2D QBMM one rank vs. multi-rank comparison
• 1D QBMM one rank vs. multi-rank comparison
• EL Bubbles simulation and post-process verification
• Verify that the existing 2MPI rank golden files are correct

Checklist

• I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature