Experimenting with re-organizing the scripts directory

[tobeycarman]

Have been needing to do this for a while, but finally made a breakthrough with understanding Python's import mechanism. The goal is to cleanup the scripts/ directory, making it easier to find the right info and easier to understand how all the pieces fit together. With this initial stab - see branch "reorganize" I chose the following layout:

scripts
├── drivers
│   ├── Sensitivity.py
│   └── __init__.py
├── tests
│   ├── doctests
│   │   ├── doctests_Sensitivity.md
│   │   ├── doctests_outspec_utils.md
│   │   ├── doctests_param_util.md
│   │   ├── doctests_qcal.md
│   │   ├── doctests_runmask_util.md
│   │   └── doctests_setup_working_directory.md
│   └── run_tests.py
├── util
│   ├── __init__.py
│   ├── input.py
│   ├── output.py
│   ├── outspec.py
│   ├── param.py
│   ├── qcal.py
│   ├── runmask.py
│   └── setup_working_directory.py
│
│  ...all the other stuff that I haven't touched yet...

With this arrangement I was able to get all the tests to run, from the main directory of the repo by doing the following:

develop@4058af50b550:/work$ PYTHONPATH="/work/scripts" python -m doctest scripts/tests/doctests/doctests_runmask.md

Really just setting the PYTHONPATH environment variable before calling the test script so that within the script, the module importing will work. This means that in order to use the utility modules in your own script you might do something like this:

# sample.py

import util.param

v = util.param.get_value(....)

And then call your script like so:

$ PYTHONPATH="/work/scripts" python sample.py

My hunch now is that setting the environment variable is preferable to using relative imports or hacking around with sys.path.

Hoping to fix a bunch of things with this re-organization:

• clean up strange script names (i.e. runmask utility has an underscore in its name making it hard to import, and some of the utility scripts were singular _util.py while others were plural _utils.py
• make the project layout more Pythonic allowing modules to be more easily re-used
• isolate the tests into their own directory so that as we add more they don't flood the scripts folder
• make the module easier to use in secondary programs, such as the MADS auto calibration tooling.

A few outstanding concerns:

• not sure what to do with the calibration directory and how much support we want to continue to give to the calibration_viewer.py
• not sure how to categorize a bunch of the remaining files in the scripts/ folder
• not sure if plotting should be folded into util directory, or kept separate (a bunch of the existing utility scripts are a mixture of plotting and utility functions)
• thinking about how to fold doctests into the Sphinx documentation
• thinking about how to get example plots auto-generated and included in Sphinx documentation...

Further discussion welcome!

[rarutter]

I think it's looking good so far. My initial inclination would be to have a plotting subdirectory and at least put the pure plotting scripts there. How much plotting is in utility scripts currently? Is it worth extracting it to new scripts or should we just point out what plotting ability there is in Discussion threads?