Testing data repo

[tobeycarman]

Problem

The testing data that we need, both for ecological testing and software testing, are too large to reasonably store in the repository. We currently have ~350MB of testing data (see testing-data/ directory) but it is easy to envision accumulating up to several GB of test data. Keeping this checked into the repository would make cloning the repo unduly expensive in many situations (both for end users and in CI/CD pipelines).

Up until Aug 2024, we have been experiment with Git-LFS for this data, but we reached the bandwidth limit and need to find a different solution.

Notes

• git-LFS, git-annex are lower level than DVC and DataLad

• DVC is machine learning specific, DataLad is geared toward researchers storing (and sharing) data

• git-annex uses symlinks. so the symlinks are tracked in version control but the data is elsewhere (S3, etc)

• DVC uses a content addressable structure - the .dvc files are tracked, and each .dvc file has a hash in it. Then the data is stored in a set of folders based on the hashes. This data can be anywhere - on your local machine, or in the cloud

• Looking at the DVC data in the cloud it is slightly confusing because it is in this content addressable storage format.

• DataLad is built on Git and Git-annex

• DVC is as much about pipelines and DAGs as it is about tracking data. It tracks and versions data, but really in the interest of building pipelines that generate and use the data.

• https://stackoverflow.com/questions/10276604/where-in-repository-to-store-test-data

• https://softwareengineering.stackexchange.com/questions/264925/where-should-i-store-test-data

[tobeycarman]

• git annex with Google Cloud Storage as a remote proved to be tricky
  • relies on another package, rclone, which also has the concept of a "remote", so the overlapping terminology is confusing
  • tried on MacOS 13.4 and was unable to get it to work
    • able to get rclone to authenticate, but unable to get git annex initremote to work: error has to do bucket and object permissions on the remote I think, though error reads:

      git-annex: this rclone remote does not support empty directories 
      failed
      initremote: 1 failed
      

  • understand here that the exporttree functionality may not be mature yet
    • exporttree allows for content to be stored on a remote in a normal file tree looking structure (with file names, etc) rather than the content addressable storage layout (sets of folders named by hash values).
    • dvc also suffers this limitation as far as I can tell.
• dvc setup with Google Cloud Storage was very easy - biggest problem is that the content stored in GCS is in the content addressable layout

[tobeycarman]

https://github.com/kevin-hanselman/dud