fix greedy batch to 10 gb

[thommodin]

This pull request introduces several enhancements and configuration changes to the src/data_index/defaults/cluster.py file, focusing on improving scalability, testability, and performance of the data indexing pipeline. The most significant changes include switching to a facility-subset-based live inventory source, updating file fetching logic, parameterizing table names for test isolation, and increasing resource allocations for Fargate clusters.

Inventory Source and Data Fetching Improvements:

• Replaced LiveS3InventorySource with LiveS3InventorySourceFacilitySubset to enable inventory scanning in facility-based subsets, improving scalability and control over the number of files processed per facility. A new subset_per_facility parameter is set to 2,000. [1] [2]
• Updated the file fetching logic to use ThresholdFileFetcher, which delegates to S5CMDFetcher or S3Fetcher based on file size, and introduced a configurable number of S5CMD workers (S5CMD_WORKERS = 8). [1] [2]

Testability and Table Management:

• Modified the run_index_cluster function to append the current Prefect flow run ID to Iceberg table names for both structured and unstructured sinks, ensuring test isolation and preventing table name collisions during concurrent test runs.
• Updated the structured metadata table name to include the schema version dynamically using StructuredMetadata.SCHEMA_VERSION.

Cluster Resource Configuration:

• Increased Fargate cluster resources: doubled the number of workers to 8, and significantly increased CPU and memory allocations for both the scheduler and workers to support larger parallel workloads.
• Set transform_max_workers default to 32 for increased parallelism in transformation tasks. [1] [2]