MPI-based program run slower and slower as execution proceeds

[spring-png]

I ran the amrex program on the supercomputing cluster and I found the it become slower and slower. The total grids didn't vary too much, but the computational time to advance the same steps become more and more. Below is part of the profile and Memprofile


I think the reasom maybe that most time was used for the communication.
But when I ran the program on an older supercomputing cluster, this issue did not occur. In the old system, the MPI initialized with thread support level 1 (perhaps the version of the MPI is old), while the current initialized with thread support level 0. So, I wonder whether the MPI support level matters? And how can I activate thread support level 1 in the current system? The version of the MPI in the current system is oneAPI/2022.1. And is there any other suggestion to deal with the issue?

[WeiqunZhang]

Maybe there is just not enough work. The high water mark of memory used by Fab is only 22 MB.

[WeiqunZhang]

And the peak number of MultiFab is 46. So it's about 0.5 MB per MultiFab.

[spring-png]

Thanks for your suggestion.
I compare the old system with the new system by runnning the same job. Below is the report for the old system


And below is the report for the new system


The time for the new system to advance the flow field is less than the old system, but more time is spent on the communication. I think this may be the problem of the communication. The mpi module used in the new system is oneAPI/2022.1. So how can I improve the communication effeciency?