Ease the linking of tasks with matching outputs/inputs

[loichuder]

Some workflows have tasks that all need a specific input (output folder, physical constant, etc.).

To achieve this without implicit coupling and overcomplicating the workflow, we pass the input from task to task to share it across the workflow. In essence, this means every link need to connect the output of the previous task to the input of the next task.

This is tractable for one or two inputs but can be overwhelming if there are more.

To alleviate that, we propose to implement an automatic mapping for tasks with matching outputs/inputs. This would be a convenience feature that can be overwritten by explicit data mapping.

The proposed implementation would use Pydantic models and there are actually two ways of doing it:

1. The specific inputs could be contained in a pydantic model as output. When passing the pydantic model to the next task, the model would be expanded to fill matching inputs
2. Alternatively, the whole Ewoks task pydantic input/output model could derive from eachother. This would create a mapping automatically, can be overwritten by explicit mapping. The automatic mapping can be based on either one of these options:
   • The downstream input model to be a super or subclass of upstream output model.
   • Output names and types (or even just the names so pydantic is not needed). To be checked if this can simply be achieved with map_all_data=True .

[maxenceprog]

i think i do not understand something with engine :

@loichuder @woutdenolf

If i want to implem that i would like to change instantiate_task_static in graph.execute.sequential OR add_dynamic_inputs in inittask

But i guess other engines do not use at all that methods as the entry point is execute_graph ?

So i cannot change this behaviour for ALL engines ? Or I miss something ?

[maxenceprog]

I'm not sure i understand the difference between 1 and 2.

For what i understand :

i should check if source outputs names (and eventually type) match dest input pydantic model.

For each match, add source input to dynamic outputs.

Also i'm not undestanding if it's

• A modification of map_all_data=True behviour
• A modification of default behaviour when map_all_data=False

[loichuder]

I'm not sure i understand the difference between 1 and 2.

In 1., the output itself (single) is a Pydantic model instance and is expanded automagically.

In 2. the Pydantic model is defining the outputs.

---

I created the issue to keep track of our discussions during the workflow meeting about "global state" (or however you want to call it) and I didn't update it with our latest discussions, my bad.

As such, I think we should discuss together before you tackle this since it is not well defined for now.

[maxenceprog]

ok maybe during workflow meeting ?

[loichuder]

Tomorrow or Friday rather

[woutdenolf]

instantiate_task_static is a helper function of the "core" engine. The ewokscore.inittask.instantiate_task function however is used by other engines.

Ideally the feature would be implemented like we implement map_all_data.

[woutdenolf]

I'm not sure i understand the difference between 1 and 2.

Me neither.

Proposal 1

The specific inputs could be contained in a pydantic model as output. When passing the pydantic model to the next task, the model would be expanded to fill matching inputs

So I understand this like

class Task1OutputSubModel(BaseModel):
    a: str
    b: int

class Task1OutputModel(BaseOutputModel):
    out1: str
    out2: Task1OutputSubModel

class Task2InputModel(BaseInputModel):
    a: str
    b: int
    c: float

So when connecting from node Task1OutputModel to Task2InputModel then ewoks would map

• task1.out2.a -> task1.a
• task1.out2.b -> task1.b

If we go this way I would not always do this, but add a flag similar to map_all_data: map_model_field=True or something. Or Task1OutputSubModel should not derive from pydantic.BaseModel but from a special model class that we create, for example EwoksMappingModel or something.

Proposal 2

the whole Ewoks task pydantic input/output model could derive from eachother. This would create a mapping automatically, can be overwritten by explicit mapping

So I understand this like

class Task1OutputModel(BaseOutputModel):
    out1: str
    a: str
    b: int

class Task2InputModel(Task1OutputModel):
    a: str
    b: int
    c: float

class Task3InputModel(Task1OutputModel):
    a: str

So when connecting from node Task1OutputModel to Task2InputModel and Task1OutputModel to Task3InputModel then ewoks would map

• task1.a -> task2.a
• task1.b -> task2.b
• task1.a -> task3.a

We currently have

from pydantic import BaseModel


class BaseInputModel(BaseModel, arbitrary_types_allowed=True):
    pass


class BaseOutputModel(BaseModel, arbitrary_types_allowed=True):
    pass

which means Task2InputModel and Task3InputModel would have BaseOutputModel in the parents which is semantically confusing. It would not be a problem currently but at some point, something might be added to the base models that would be different for BaseInputModel vs. BaseOutputModel.

Perhaps this puts in question the presence of these base models.

[maxenceprog]

Thank you @loichuder and @woutdenolf . I spoke with Loic this afternoon, i will try to sum up the discussion here today but the conclusion was there is no perfect method to solve problem