Feature/add structured metadata

[thommodin]

This pull request introduces several important improvements and clarifications to the local subset metadata analysis workflow, S3 table configuration, and domain terminology. The main themes are: (1) clarifying the canonical use of "Facility" vs "collection" in domain language and code, (2) adding new local subset and analysis handoff documentation, (3) updating configuration and defaults for local and subset indexing, and (4) providing new developer documentation for S3 table management.

Domain terminology and schema clarifications:

• Updated CONTEXT.md to clarify that "Facility" is now the canonical term for the second S3 path segment (e.g., IMOS/ANMN/NSW/file.nc), while collection is retained only as a legacy field/schema name for backward compatibility. All relevant glossary, ambiguity, and schema sections were updated to reflect this, and ambiguity between "collection" and "facility" is now explicitly resolved. [1] [2] [3] [4]

Local subset pipeline and configuration:

• Added a new module local_subset.py that defines a local subset pipeline using LiveS3InventorySourceFacilitySubset, with explicit configuration for inventory, sinks, and a threshold-based fetcher. This enables targeted facility-based sampling for local analysis.
• Refactored local.py to use the canonical AttributeNetCDFExtractor by default, updated variable names for clarity, and tuned worker/thread settings for local runs. [1] [2]

Analysis and handoff documentation:

• Added two new handoff documents:
  • 2026-06-02-local-subset-metadata-analysis.md: Outlines the analysis design, metrics, glossary decisions, and concrete next steps for evaluating facility extraction, structured field coverage, and unstructured mapping correctness.
  • 2026-06-02-structured-metadata-schema-feedback.md: Documents stakeholder feedback and decisions for upcoming schema changes, including new fields, alias handling, and test requirements.

Developer documentation and analysis support:

• Added a new section to managing-s3-tables.md documenting how to delete S3 tables and table buckets using the AWS CLI, supporting easier management of Iceberg tables.
• Added a new analysis notebook (analysis/structured_metadata/v1.ipynb) to demonstrate configuration and usage of Iceberg table configs for structured and unstructured metadata.
• Added new analysis dependencies (genson, natsort) to pyproject.toml to support schema generation and sorting in analysis scripts.

Summary of most important changes:

1. Domain terminology and schema updates

• Clarified "Facility" as the canonical term for the second S3 path segment in all documentation and code references; "collection" is now only a legacy field/schema name. [1] [2] [3] [4]

2. Local subset pipeline and configuration

• Added local_subset.py for facility-based local subset indexing, with explicit configuration for inventory, sinks, and fetcher.
• Refactored local.py to default to AttributeNetCDFExtractor, improved naming, and tuned worker/thread settings for local development. [1] [2]

3. Analysis and handoff documentation

• Added handoff docs for local subset analysis and structured metadata schema feedback, capturing decisions, metrics, and next steps for analysis and schema evolution. [1] [2]

4. Developer and analysis support

• Added documentation for deleting S3 tables and table buckets with AWS CLI.
• Introduced a new analysis notebook and updated analysis dependencies for schema and sorting support. [1] [2]