Feature/discriminated implementations

[thommodin]

This pull request introduces several improvements and refactorings to the data indexing codebase, focusing on standardizing model type identification, improving orchestration reliability, and enhancing task runner configuration. The most significant changes are the addition of explicit type fields to key Pydantic models, a switch to Dask for local task execution, and improved orchestration startup logic.

Model Standardization and Identification:

• Added a type field (using typing.Literal) to all major Pydantic models—including fetchers, inventory sources, extractors, and sinks—to enable explicit model type identification and easier serialization/deserialization. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]

Task Runner and Dependency Updates:

• Switched the local subset indexing flow to use DaskTaskRunner instead of ThreadPoolTaskRunner, and added the required prefect_dask import. [1] [2]
• Added pydantic to the project dependencies in pyproject.toml to ensure compatibility with new model features.

Orchestration and Cluster Warmup Improvements:

• Enhanced the orchestrate function to explicitly warm up the task runner before dispatching ETL batches, improving reliability and logging. [1] [2]
• Changed batch submission logic to stream results as batches complete, rather than waiting for all batches before processing results. [1] [2]

Configuration and Defaults:

• Added skip_cleanup configuration option to PrefectFargateClusterConfig for controlling resource cleanup behavior.
• Updated default cluster configuration with explicit execution_role_arn and task_role_arn values.

Miscellaneous Refactoring:

• Added missing typing imports to several files to support new type annotations. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
• Minor cleanup of unused imports and variables.

These changes collectively improve the maintainability, configurability, and robustness of the data indexing system.