Update _demote_entry to append tier latency timings#832
Conversation
|
MLCommons CLA bot: |
|
I've taken note of the MLCommons CLA comment. To clarify, I've already signed the MLCommons CLA and submitted my GitHub ID to the onboarding form. You can check the status by retriggering the bot on this pull request. I'm willing to wait for the necessary authorization before this PR can be merged. Once confirmed, I'll proceed with the necessary steps. I'd like to point out that, as a contributor, I'm following all the established guidelines for the project. I've reviewed the code, fixed a specific issue, and made sure to follow the project's code style and conventions. I'm committed to ensuring the quality of my contributions, and I'm happy to address any concerns you may have. Let's work together to improve the project and maintain its high standards. |
This PR fixes a critical issue in the _demote_entry function of kv_cache/cache.py where the resulting timings are not being appended to the per-tier latency lists. This omission leads to inaccurate latency P95 SLA checks, which can have significant consequences for our system's performance and reliability. Upon reviewing the code, I realized that the timings are not being appended due to a missing append operation. I verified this fix by running the existing test suite and manually testing the _demote_entry function to ensure the timings are correctly appended to the latency lists. With this fix, we can expect improved accuracy of our latency P95 SLA checks, which will help us better understand and optimize our system's performance.