What happens
uv run poe benchmark-needs (or pytest tests/benchmarks) fails 12 of the 18 tests in
tests/benchmarks/test_querying.py:
FAILED tests/benchmarks/test_querying.py::test_filter_single_need_simple[100]
…
FAILED tests/benchmarks/test_querying.py::test_filter_needs_and_parts_complex_eval[1000]
E AttributeError: Mock object has no attribute 'variant_data_proxy'.
Did you mean: 'variant_data_file'?
12 failed, 6 passed
Reproduce quickly (no timing) with:
uv run --frozen --group benchmark pytest tests/benchmarks/test_querying.py --benchmark-disable -q
Why
test_querying.py builds its config as Mock(spec=NeedsSphinxConfig) (line 113). A spec'd
mock only has the attributes the class declares, and variant_data_proxy is not a field of
NeedsSphinxConfig: sphinx_needs/needs.py assigns it dynamically in
_derive_variant_data_proxy() (config.variant_data_proxy = …), and the filter code reads it
(filter_common.py, directives/needif.py). The dynamic assignment arrived with
needs_variant_data in #1715 (2026-05-26); the benchmark test has not changed since #1677
(2026-03-31), so the suite has been red since then.
CI does not see it: the benchmark workflow runs the directory twice with -k _time and
-k _memory, which together collect 3 of the 25 tests, all outside the failing set. It was
found by running every poe task on the workspace-layout branch (#1839), where the failure
reproduces identically on the moved tree and on master.
Possible fixes
- Declare
variant_data_proxy on NeedsSphinxConfig (it is a derived value, so a field with
a None default that _derive_variant_data_proxy fills), which also makes it visible to the
type checker instead of being an untyped attribute; or
- keep it dynamic and have the benchmark fixture set it on the mock
(config.variant_data_proxy = None).
Separately: should the benchmark job collect the whole directory, so the non-timed tests in
it are at least executed? Today a benchmark that stops working is invisible until someone
runs the task locally.
What happens
uv run poe benchmark-needs(orpytest tests/benchmarks) fails 12 of the 18 tests intests/benchmarks/test_querying.py:Reproduce quickly (no timing) with:
Why
test_querying.pybuilds its config asMock(spec=NeedsSphinxConfig)(line 113). A spec'dmock only has the attributes the class declares, and
variant_data_proxyis not a field ofNeedsSphinxConfig:sphinx_needs/needs.pyassigns it dynamically in_derive_variant_data_proxy()(config.variant_data_proxy = …), and the filter code reads it(
filter_common.py,directives/needif.py). The dynamic assignment arrived withneeds_variant_datain #1715 (2026-05-26); the benchmark test has not changed since #1677(2026-03-31), so the suite has been red since then.
CI does not see it: the benchmark workflow runs the directory twice with
-k _timeand-k _memory, which together collect 3 of the 25 tests, all outside the failing set. It wasfound by running every poe task on the workspace-layout branch (#1839), where the failure
reproduces identically on the moved tree and on
master.Possible fixes
variant_data_proxyonNeedsSphinxConfig(it is a derived value, so a field witha
Nonedefault that_derive_variant_data_proxyfills), which also makes it visible to thetype checker instead of being an untyped attribute; or
(
config.variant_data_proxy = None).Separately: should the benchmark job collect the whole directory, so the non-timed tests in
it are at least executed? Today a benchmark that stops working is invisible until someone
runs the task locally.