is it possible to keep monitoring when job errors?

[thangckt]

Developers,

Thank you so much for a useful packages.
I want to ask for improve jobs handling.

The current situation

When we submit jobs with this snippet

submission = Submission(work_base="./",
                            machine = machine,
                            resources = resources,
                            task_list = task_list,
                            ...)
submission.run_submission() 

Assume we have 50 jobs in task_list, if there is one of them occur runtime error, then dpdispatcher immediately exist the connect session, and left other 49 unfinished jobs uncontrolable, we then have to handle them manually. This is inefficient.

My questions

1. Is it possible to keep the connect session and keep monitoring unfinished jobs until all finished or all errors. Then just download the finished results?

2. Or, is it possible to postpone the report of error jobs whenever all other finished jobs.

3. is it possible to disable reentry error jobs? Some jobs takes very long time before error occurs, reentry 3 times as in current code is likely wasting computing resource.

Thanks.

[njzjz]

It seems that you just need to append || : to command, which can cheat the dpdispatcher into thinking that the jobs have been finished even if they fail. (This approach has been documented.)

[thangckt]

@njzjz, thank you for information,

But, that approach should not be used. There will no any fail jobs, and eventually leads to error file not found for downloading.

What I expect is: delay the report of fail jobs until all other jobs finished (not treating fail jobs as finished jobs)

[thangckt]

I solve the point 3 in PR #545