Added parquet support for filters parameter

[gtorrini]

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes
• (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

Notes

• Missing release note (no ticket associated yet).
• I didn't do any semantic validation on the filters, as this is performed by filters_to_expression early on in ParquetDataset initialization.
• I tried to get started on some basic unit test checks, but it seems there's a wrapper/except block preventing the ValueError for invalid columns in the filter predicates from being interpreted directly. See the #FIXME comments in test_parquet.py. The get call outside the assertRaises context manager DOES raise a ValueError, but the get call inside the assertRaises context manager raises an AssertionError for not raising a value error.

====================================== short test summary info =======================================
FAILED test_parquet.py::ParquetFormatterDataFrameTestCase::testMultiIndexDataFrame - ValueError: Failure from formatter 'lsst.daf.butler.formatters.parquet.ParquetFormatter' for data...
FAILED test_parquet.py::InMemoryDataFrameDelegateTestCase::testSingleIndexDataFrame - AssertionError: ValueError not raised

[codecov]

Codecov Report

❌ Patch coverage is 20.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.46%. Comparing base (5a7fd38) to head (1280ff0).
⚠️ Report is 186 commits behind head on main.

Files with missing lines	Patch %	Lines
python/lsst/daf/butler/formatters/parquet.py	20.00%	11 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1308      +/-   ##
==========================================
- Coverage   89.48%   89.46%   -0.03%     
==========================================
  Files         367      367              
  Lines       49852    49867      +15     
  Branches     5855     5863       +8     
==========================================
+ Hits        44611    44614       +3     
- Misses       3853     3864      +11     
- Partials     1388     1389       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[erykoff]

Is the indentation correct above? This won't do the check if it is an Expression; is that intended?

[gtorrini]

It is not, I should fix it to perform the check if it is an Expression as well.

[erykoff]

I don't understand this section at all. If it's a list of tuples you convert a list to a list of lists? And then if it's a list of lists it raises?

[gtorrini]

I converted the list of tuples to a list of list of tuples for streamlined column checking later on (par_filters in DNF). It should only raise the TypeError if it is not a list of tuples or list of lists.

[erykoff]

Tests of the functionality is missing.

[gtorrini]

Yes, this is addressed in the third bullet points of the notes above. I couldn't figure out why it raises a ValueError outside of the context manager, but calling the context manager to check for a ValueError raises an AssertionError.

[gtorrini]

After revisiting Eli's feedback and doing some more research, I think implementing parquet filter support is probably above my pay grade:

• As far as I can tell, PyArrow does not perform any dataset-aware validation on the filters before reading; it only semantically validates predicates. See ParquetDataset initialization, filters_to_expression, and ParquetDataset.read method for more info.
• In addition, I could not find a public API exposing field references in a pyarrow.dataset.Expression, which prevents me from validating against the fields in the schema.

Passing filters without any field checks to pq.read_table--even if they are a properly formatted ds.Expression, List[Tuple], or List[List[Tuple]]--could result in some very late errors raised by PyArrow. I'm not sure what degree of validation is preferred at the daf_butler API boundary, so I'll defer to someone with more expertise.