Timx 638 publish index doc count metric to cloudwatch

[jonavellecuerdo]

Purpose and background context

When the bulk-update or reindex-source CLI commands are executed, we want to understand the result of performed bulk actions on a given OpenSearch index. Publishing a metric (e.g., number of documents in an index) to CloudWatch will allow us to monitor the health of our OpenSearch indexes over time.

At high-level, here are the key changes to review:

• Introduce utils for publishing metrics to CloudWatch
  • As mentioned in previous discussions with X, this represents a much simplified version of the MetricsClient from DSC. This simplification is in congruence with our simple use case of CloudWatch metrics for TIM: we are only publishing a single metric to CloudWatch towards the end of a bulk-update or reindex-source CLI command execution.
• Retry decorator supports optional until_condition
  • If an until_condition is provided and the decorated function returned a value (i.e., get_index_doc_count()), the method will now also check if the provided condition is met; otherwise, retry until timeout, unexpected error or condition is met.
• TIM bulk-update CLI command publishes an 'Index Doc Count' metric
  • First, TIM will determine the initial document count for the index
    • Note: The get_index_doc_count() method is wrapped in a retry without an until_condition. At this point, no bulk actions have been performed so we can expect count to be accurate.
  • After bulk deletes are performed, TIM will attempt to publish the 'Index Doc Count' metric.
    • If it was able to get the initial document count at the beginning, it will set an until_condition so that the retry will query counts until it is equal to initial count - deleted count
    • Otherwise, it will retrieve an index doc count, logging a warning that it may be inaccurate.
• TIM reindex-source CLI command publishes an 'Index Doc Count' metric
  • This is much simpler than the former! 🎉 Because reindex-source creates a whole new index, the until_condition for the retry is to confirm that the count == created count

How can a reviewer manually see the effects of these changes?

Access the view above by visiting CloudWatch Metrics in Dev1.

1. On the right menu bar, select Metrics -> Classic Metrics (guess it recently updated 👀 )
2. Select Timdex namespace.
3. Select index dimension.
4. From the list of indexes, select the primary indexes for each source (i.e., what's shown in the screenshot)
5. On the top-left corner, you can choose how to view the metrics--Data table was selected for the screenshot above.

It is likely we will have more ideas on how best to organize these metrics (i.e., add new "dimensions"), but this is a start! :)

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES-ish: This may increase the duration of our TIM runs.

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-638

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[Copilot]

Pull request overview

This PR adds CloudWatch metric publishing to track OpenSearch index document counts after bulk-update and reindex-source CLI runs, and extends the existing retry decorator to support retrying until a returned-value condition is met.

Changes:

• Introduces a small CloudWatch metrics utility (CloudWatchMetricsClient, Metric) and exports it via tim.utils.aws.
• Adds helpers.retry(..., until_condition=...) support and uses it when querying index doc counts for metric publication.
• Adds OpenSearch helpers (get_index_doc_count, refresh_index) and includes status codes in BulkActionError messages; updates tests accordingly.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tim/utils/aws/cloudwatch_metrics.py	Adds CloudWatch metric publishing client + Metric dataclass.
tim/utils/aws/init.py	Exposes CloudWatch metrics utilities from the aws utils package.
tim/utils/init.py	Adds utils package initializer.
tim/opensearch.py	Adds index doc count + refresh helpers; includes status in raised bulk errors.
tim/helpers.py	Extends retry decorator with optional until_condition.
tim/errors.py	Updates BulkActionError to include HTTP status in the exception message.
tim/cli.py	Publishes “Index Doc Count” metric after bulk-update and reindex-source.
tests/test_opensearch.py	Updates tests around bulk ops (adds patching blocks).
tests/test_helpers.py	Adjusts retry tests (removes a freeze_time decorator).
tests/test_cli.py	Patches metric publishing in CLI tests; updates BulkActionError construction.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ehanson8]

Looks good to me and it's obviously publishing metrics in Dev1, great work! 2 optional suggestions