ADIOS2's HDF5 Engine to support OpenPMD

[tybulut]

Remember to motivate and summarize considering the community. Provide enough information to motivate such as:

Why is this feature important?
OpenPMD standard: https://github.com/openPMD/openPMD-standard/blob/latest/STANDARD.md

It will be great to have ADIOS2's HDF5 engine output is compliant with OpenPMD for wider adoption.

What is the potential impact of this feature in the community?
HDF5 outputs created by ADIOS2 with setting engine = HDF5 could be read by openPMD compliant readers.

Is your feature request related to a problem? Please describe.
HDF5 outputs created by ADIOS2 with setting engine = HDF5 cannot be read by openPMD compliant readers because it prepends the keys with "/Step0" where 0 is the step number.

Describe the solution you'd like and potential required effort

void HDF5Common::StaticGetAdiosStepString(std::string &stepName, size_t ts)
{                                                     
    stepName = "/Step" + std::to_string(ts);
}   

Is it possible to introduce a configuration option called "openPMD_compatible" or something similar such that when it is set to true this is calculated as:

void HDF5Common::StaticGetAdiosStepString(std::string &stepName, size_t ts)
{                                                     
    if (...)
        stepName = "/data/" + std::to_string(ts) + "/";
    else
        stepName = "/Step" + std::to_string(ts);
}   

Effort seems small.

Describe alternatives you've considered and potential required effort
Alternative is to make the prefix "/Step" to be fully customizable instead of using a setting such as openPMD_compatible setting. However, this could increase the effort.

Additional context
None

Note: contents adapted from the VTK contribution guidelines

[guj]

Thanks for the interest!
The most effective way to ensure your data files comply with the openPMD standard is by utilizing the openPMD-api interface, available at https://github.com/openPMD/openPMD-api. It supports HDF5, ADIOS and JSON formats.
There are examples and tests in the repository. Let me know if you have questions.

[tybulut]

Thank you. But the definition of "most effective" is relative, I guess. :)

• For projects that already adopted adios2 directly and want to leverage its built-in HDF5 integration, injecting the openPMD-api library will be a large investment.
• This also violates the assumption of extensibility of adios2 and requires to adopt a library above the adios2 abstraction to get some capability that could have been implemented down in the stack while maintaining the Adios2 API as an abstraction.

PS: I am happy to provide a patch if I can get help reviewing my design and patch by Adios2 contributors.

[guj]

Thank you. But the definition of "most effective" is relative, I guess. :)

• For projects that already adopted adios2 directly and want to leverage its built-in HDF5 integration, injecting the openPMD-api library will be a large investment.

Looks like you have particle mesh data, and wanting to use both ADIOS and HDF5 formats to store, and complying with the openPMD standard is the end goal, possibly leveraging the existing openPMD-api work will be the easier option in the long run.

• This also violates the assumption of extensibility of adios2 and requires to adopt a library above the adios2 abstraction to get some capability that could have been implemented down in the stack while maintaining the Adios2 API as an abstraction.

PS: I am happy to provide a patch if I can get help reviewing my design and patch by Adios2 contributors.

Sure I can review it.