Wrap executorlib executors

[liamhuber]

To exploit the new caching interface provided in executorlib-1.5.0 so that nodes can rely on their running state and lexical path to access previously executed results.

Locally with the SingleNodeExecutor everything is looking good, but that doesn't natively support terminating the process that submit the job. I'd like to play around with this using the SlurmClusterExecutor on the cluster before making further changes.

[github-actions]

👈 Launch a binder notebook on branch pyiron/pyiron_workflow/executor

[codecov]

Codecov Report

Attention: Patch coverage is 88.88889% with 9 lines in your changes missing coverage. Please review.

Project coverage is 92.05%. Comparing base (4841e34) to head (f579159).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pyiron_workflow/node.py	70.00%	6 Missing ⚠️
pyiron_workflow/executors/wrapped_executorlib.py	96.29%	1 Missing ⚠️
pyiron_workflow/mixin/run.py	96.42%	1 Missing ⚠️
pyiron_workflow/nodes/composite.py	66.66%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (88.88%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #678      +/-   ##
==========================================
- Coverage   92.11%   92.05%   -0.07%     
==========================================
  Files          33       34       +1     
  Lines        3665     3725      +60     
==========================================
+ Hits         3376     3429      +53     
- Misses        289      296       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bot spam

[liamhuber]

This is currently working to all attempted behaviour when run together with pyiron/executorlib#712

[liamhuber]

ImportError: cannot import name 'TestClusterExecutor' from 'executorlib.api' (/home/runner/work/pyiron_workflow/pyiron_workflow/cached-miniforge/my-env/lib/python3.13/site-packages/executorlib/api.py). Did you mean: 'FluxClusterExecutor'?

Waiting on executorlib-1.5.3

[liamhuber]

Together with pyiron/executorlib#732 this is working very nicely on the cluster now. I can

• Let the workflow run
• Interrupt the workflow after the slurm job has started
  • Restart the workflow after the slurm job is complete
  • Restart the workflow while the slurm job is still going

and in all cases everything runs perfectly smoothly. I.e. I can start with this:

import pyiron_workflow as pwf
from pyiron_workflow.executors.wrapped_executorlib import CacheSlurmClusterExecutor


wf = pwf.Workflow("executor_test")
wf.n1 = pwf.std.UserInput(20)
wf.n2 = pwf.std.Sleep(wf.n1)
wf.n3 = pwf.std.UserInput(wf.n2)

wf.n2.executor = (CacheSlurmClusterExecutor, (), {"resource_dict": {"partition": "s.cmfe"}})

wf.run()

And then either let it run, or restart the kernel and follow up with this after the appropriate delay for the case I'm interested in:

import pyiron_workflow as pwf
from pyiron_workflow.executors.wrapped_executorlib import CacheSlurmClusterExecutor

wf = pwf.Workflow("executor_test")
wf.load(filename=wf.label + "/recovery.pckl")

wf.failed = False
wf.use_cache = False
wf.run()

Outside the scope of this PR but on the TODO list is:

• Convenience around use_cache and Workflow use_cache is annoying on composites #699
• Convenience around running workflows in a background thread so I can make a real save and not rely on a recovery file Let workflows be easily run in the background #700
• CI tests on Slurm ala Test with SLURM executorlib#726 (done and merged in Draft a slurm test #704)
• Adding the new wrapped executorlib.SlurmClusterExecutor to the API Expose executorlib SLURM power in API #708
• Removing the old way of finding serialized slurm results Remove old caching tools #709

[liamhuber]

Codecov complains that Node._clean_wrapped_executorlib_executor_cache is not being tested, but it's just flat out wrong. It is absolutely being invoked in tests.integration.test_wrapped_executorlib.TestWrappedExecutorlib.test_automatic_cleaning.

[liamhuber]

• CI tests on Slurm

Actually, I'd be more comfortable with this PR if it included these. I'll still leave exposure in the API for later, but let's take a crack at robust testing right here.