Replace task executor, split workflow

[javiermtorres]

What's changing

This PR allows a workflow to be specified as a list of jobs to be run sequentially:

    workflow_payload = WorkflowJobsCreateRequest(
        name=workflow_name,
        description="Test workflow for inf and eval",
        experiment_id=experiment_id,
        dataset=str(dataset_id),
        model=model, 
        job_timeout_sec=1000,
        task_definition={"task": "summarization"},
        job_list=[
            JobCreate(
                name=f"{workflow_name}-inference",
                max_samples=1,
                job_config=JobInferenceConfig(
                    model=model,  # later: {"$ref": "#/model"}
                    provider="hf",
                    task_definition=task_definition,
                    # we store the dataset explicitly below, so it gets queued before eval
                    store_to_dataset_suffix="predictions",
                ),
            ),
            JobCreate(
                name=f"{workflow_name}-evaluation",
                # max_samples=?,
                job_config=JobEvalConfig(),
            ),
        ],
    )

Currently, each job will try to use the dataset produced by the previous job (if any), and the workflow will merge all results together in a single result. Tasks are handled using asyncio Tasks and stored in a dict within their appropriate service (jobs or workflows) using their created ID as a key. The dict should do auto cleanup when the Task completes.

It currently needs support from the UI to be used there, but auto tests should work ok.

Closes #774

How to test it

Run the auto tests and follow the job information in Ray and MLFlow. Everything should be stored in their place.

Additional notes for reviewers

N/A

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required