New features: chunking and compression

[khou2020]

This PR is to add data chunking and compression features. By using additional metadata and manipulate the space between data object file offsets, these features can be developed without breaking the NetCDF file format specifications.

More information about the design and implementation are described in K. Hou, Q. Kang, S. Lee, A. Agrawal, A. Choudhary, W. Liao. Supporting Data Compression in PnetCDF, published in the International Conference on Big Data, 2021.

An example program is available in ./examples/C/chunk_compress.c

[gsjaardema]

This sounds very interesting and is in line with some work we are planning to do this fiscal year (plan was for using the HDF5 compression under netCDF). If this is planned to go into production in the short term, we would also include this in the work that we do.

[wkliao]

Since you mentioned HDF5, I wonder if there is any thing missing in HDF5 but is in this PR of PnetCDF.

[gsjaardema]

No, we basically want to give the users some options. Some prefer HDF5-based files and some prefer the native based files. It also gives us flexibility if having problems we can try the other format and see if that does/doesn't have a similar issue.

Dealing with the compression filter plugins on a netCDF-4 file is somewhat problematic since a user needs to know what was used to write the file at the time it is read to make sure the correct filter exists. That is our main delay in implementing some of the compression filters is that we can't guarantee that a reader of the file will have the correct filter or even given a raw hdf5 file, how do we know what filter is needed to read it...

[wkliao]

That is indeed a challenge.

Another thing I learned is users may want to use different compression
parameters for different variables. Is this supported in your case?

[gsjaardema]

Another thing I learned is users may want to use different compression parameters for different variables. Is this supported in your case?

We have not investigated that yet. Currently using the same parameters for all datasets. We do have some integer and some double-precision floating point data sets, so different algorithms might help there, but we haven't looked at it yet.

[wkliao]

If you plan to give this PR a try, I very much welcome your feedback.

FYI. PnetCDF also supports compression/decompression in its nonblocking
APIs, which allow to compress/decompress multiple variables in one
call to ncmpi_wait_all. More information and timing results can be found in
our 2021 paper mentioned above.

[dqwu]

@wkliao
Have you tested this PR with --enable-sz --with-sz=/sz/install/path?

It looks like the SZ headers are installed under
/sz/install/path/include/sz
rather than directly in
/sz/install/path/include.

Because of that, the following check fails to find sz.h:

if test "x${have_sz}" = xyes; then
   AC_CHECK_HEADERS([sz.h], [], [have_sz=no])
fi

A possible fix would be to adjust the include path:

if test "x${SZ_INSTALL}" != x ; then
   CPPFLAGS+=" -I${SZ_INSTALL}/include/sz"
   ...
fi

[wkliao]

For such unusually installation, I suggest to set CPPFLAGS environment variable
to the SZ include path when running configure. Please let me know if that works.

[dqwu]

For such unusually installation, I suggest to set CPPFLAGS environment variable to the SZ include path when running configure. Please let me know if that works.

Actually that is the default installation of SZ.

Setting CPPFLAGS environment variable to the SZ include path before calling configure works:
export CPPFLAGS="-I /sz/install/path/include/sz"

[dqwu]

@wkliao
The following example runs correctly with the PnetCDF master branch, but hangs at the ncmpi_wait_all call when tested with this PR. Could you please take a look?

#include <mpi.h>
#include <pnetcdf.h>
#include <stdio.h>

#define DIM_LEN 8

int main(int argc, char **argv)
{
    int ncid, dimid, varid, rank;
    int vals[DIM_LEN] = {-1, -2, -3, -4, -5, -6, -7, -8};
    MPI_Offset start, count;
    MPI_Info info;
    int req;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    MPI_Info_create(&info);
    MPI_Info_set(info, "nc_chunking", "enable");

    ncmpi_create(MPI_COMM_WORLD, "test.nc", NC_CLOBBER, info, &ncid);
    MPI_Info_free(&info);

    ncmpi_def_dim(ncid, "x", DIM_LEN, &dimid);
    ncmpi_def_var(ncid, "var", NC_INT, 1, &dimid, &varid);

    ncmpi_enddef(ncid);

    if (rank == 0)
    {
        start = 0;
        count = DIM_LEN;
        ncmpi_iput_vara_int(ncid, varid, &start, &count, vals, &req);
    }
    else
        req = NC_REQ_NULL;
        
    printf("rank = %d, before ncmpi_wait_all\n", rank); fflush(stdout);
    ncmpi_wait_all(ncid, 1, &req, NULL);
    printf("rank = %d, after ncmpi_wait_all\n", rank); fflush(stdout);

    ncmpi_close(ncid);

    MPI_Finalize();

    return 0;
}

Notes:

• The hang does not occur if nc_chunking is set to "disable".
• The hang does not occur if ncmpi_wait_all is called with NC_REQ_ALL, e.g.:
  ncmpi_wait_all(ncid, NC_REQ_ALL, NULL, NULL);

[wkliao]

Hi, @dqwu

Thanks. I was able to reproduce the error.
I pushed a fix. Please let me know if it fixes the problem.

[dqwu]

Hi, @dqwu

Thanks. I was able to reproduce the error. I pushed a fix. Please let me know if it fixes the problem.

The problem has been fixed in the latest feature branch, thanks.