Skip to content

Implement error-bounded lossy compression method MacaqueV - #341

Merged
skejserjensen merged 15 commits into
ModelarData:mainfrom
aabduvakhobov:macaque
Sep 3, 2025
Merged

Implement error-bounded lossy compression method MacaqueV #341
skejserjensen merged 15 commits into
ModelarData:mainfrom
aabduvakhobov:macaque

Conversation

@aabduvakhobov

@aabduvakhobov aabduvakhobov commented Aug 4, 2025

Copy link
Copy Markdown
Contributor

This PR implements MacaqueV, a lossy error-bounded single floating point compression method, as a replacement for Gorilla model type. Macaque adds two value preprocessing methods.
First, MacaqueV rewrites a value by 1) replacing it with the previous one within the error bound or 2) rewriting the least mantissa bits of the value within the error bound so that Gorilla uses fewer bits for encoding. Then the value is encoded using the modified version of Gorilla. Gorilla's flag bits 0 and 10 are swapped to achieve better compression when used alongside PMC-Mean and Swing.
Experiments using real-life datasets showed that MacaqueV improves ModelarDB's compression by 1.5-2.5x.

@aabduvakhobov

Copy link
Copy Markdown
Contributor Author

Hi. Although MacaqueTS (according to an ICDE 2026 paper) was already implemented earlier, please let me know if it should also be reflected in the comments. At the moment, only MacaqueV is reflected in this PR.

@skejserjensen
skejserjensen requested a review from Copilot August 5, 2025 06:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR replaces the Gorilla compression model with MacaqueV, an error-bounded lossy compression method for single floating point values. MacaqueV extends Gorilla by adding value preprocessing methods and optimizing flag bits for better compression when used with PMC-Mean and Swing models.

  • Replaces all references from Gorilla to MacaqueV across the codebase
  • Implements MacaqueV with enhanced value rewriting capabilities using error bounds
  • Modifies flag bit encoding by swapping Gorilla's flag bits 0 and 10 for better compression

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
crates/modelardb_compression/src/types.rs Updates imports and method calls to use MacaqueV instead of Gorilla
crates/modelardb_compression/src/models/mod.rs Renames module and constants from gorilla to macaque_v, adds lossless compression helper
crates/modelardb_compression/src/models/macaque_v.rs Complete implementation of MacaqueV with value rewriting and optimized flag bits
crates/modelardb_compression/src/compression.rs Updates all references and test cases to use MacaqueV instead of Gorilla
Comments suppressed due to low confidence (2)

crates/modelardb_compression/src/models/macaque_v.rs:90

  • There is a grammatical error in the comment. 'Returns true is compression' should be 'Returns true if compression'.
        }

crates/modelardb_compression/src/models/macaque_v.rs:329

  • [nitpick] Function name 'rewrite_bits_by_n' is unclear. Consider renaming to 'clear_least_significant_bits' or 'mask_lower_bits' to better describe what the function does.
fn rewrite_bits_by_n(bits_to_rewrite: u32, left_shift_by: i32) -> u32 {

Comment thread crates/modelardb_compression/src/models/macaque_v.rs
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
@skejserjensen
skejserjensen requested a review from CGodiksen August 5, 2025 06:17
@skejserjensen

Copy link
Copy Markdown
Contributor

@aabduvakhobov thank you very much for the PR. I have asked @CGodiksen to review the code first as he can provide a fresh perspective. In the meantime, please correct the issues found by ast-grep and co-pilot. Be aware that co-pilot is not always correct, but occasionally identifies legitimate issues. Finally, when creating a PR that improves an aspect of ModelarDB instead of simply providing a feature, please include metrics to indicate how big an improvement the changes provide so the trade-off between added code complexity and the resulting improvements can be evaluated.

Comment thread crates/modelardb_compression/src/compression.rs Outdated
Comment thread crates/modelardb_compression/src/models/mod.rs Outdated
Comment thread crates/modelardb_compression/src/types.rs Outdated
Comment thread crates/modelardb_compression/src/types.rs
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
@skejserjensen
skejserjensen requested a review from CGodiksen August 8, 2025 22:46
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
Comment thread crates/modelardb_compression/src/models/mod.rs
Comment thread crates/modelardb_compression/src/models/timestamps.rs Outdated
Comment thread crates/modelardb_compression/src/models/timestamps.rs Outdated

@skejserjensen skejserjensen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still a few comments that require changes to the code.

Comment thread crates/modelardb_compression/src/models/macaque_v.rs Outdated
@skejserjensen
skejserjensen merged commit 8b72034 into ModelarData:main Sep 3, 2025
4 checks passed
@skejserjensen

Copy link
Copy Markdown
Contributor

@aabduvakhobov the PR has now been merged. Thank you very much for your contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants