Job service: fix bug in getting incomplete jobs that Ray no longer knows about

[peteski22]

What's changing

• Improved granularity of error handling for get_upstream_job_status
• Added tests for get_upstream_job_status
• Added new JobStatus which indicates a job is not recoverable (i.e. it was running but now Ray no longer knows of its existence)
• Changed get_job to use get_upstream_job_status instead of the Ray client directly
• Minor renaming to indicate that the class fields are private
• Frontend tweaks to use job status rather than workflow status for annotation jobs

NOTE: This will appear in the Lumigator server log when the job can no longer be found in Ray, and prevent errors escaping into the UI via the API. No further attempts to contact Ray for status updates of that job will be made.

Closes #1296

How to test it

Steps to test the changes:

For folks who have a database already in this situation:

1. make local-up
2. Start some jobs/experiments (don't let them finish)
3. make local-down and make clean-all to remove everything (except the database it seems, which is good for this test)
4. make local-up and visit the UI .. we shouldn't be getting the wild logs in the server now

Additional notes for reviewers

Although we're adding a new JobStatus, it will never be surfaced via the API as the current logic only returns jobs which have a presence in Lumigator's storage AND in Ray (merged).

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required