Skip to content

🐛 Benchmark suite: test_querying.py mocks NeedsSphinxConfig without variant_data_proxy (12 failures CI never collects) #1840

Description

@chrisjsewell

What happens

uv run poe benchmark-needs (or pytest tests/benchmarks) fails 12 of the 18 tests in
tests/benchmarks/test_querying.py:

FAILED tests/benchmarks/test_querying.py::test_filter_single_need_simple[100]
…
FAILED tests/benchmarks/test_querying.py::test_filter_needs_and_parts_complex_eval[1000]
E   AttributeError: Mock object has no attribute 'variant_data_proxy'.
    Did you mean: 'variant_data_file'?
12 failed, 6 passed

Reproduce quickly (no timing) with:

uv run --frozen --group benchmark pytest tests/benchmarks/test_querying.py --benchmark-disable -q

Why

test_querying.py builds its config as Mock(spec=NeedsSphinxConfig) (line 113). A spec'd
mock only has the attributes the class declares, and variant_data_proxy is not a field of
NeedsSphinxConfig: sphinx_needs/needs.py assigns it dynamically in
_derive_variant_data_proxy() (config.variant_data_proxy = …), and the filter code reads it
(filter_common.py, directives/needif.py). The dynamic assignment arrived with
needs_variant_data in #1715 (2026-05-26); the benchmark test has not changed since #1677
(2026-03-31), so the suite has been red since then.

CI does not see it: the benchmark workflow runs the directory twice with -k _time and
-k _memory, which together collect 3 of the 25 tests, all outside the failing set. It was
found by running every poe task on the workspace-layout branch (#1839), where the failure
reproduces identically on the moved tree and on master.

Possible fixes

  • Declare variant_data_proxy on NeedsSphinxConfig (it is a derived value, so a field with
    a None default that _derive_variant_data_proxy fills), which also makes it visible to the
    type checker instead of being an untyped attribute; or
  • keep it dynamic and have the benchmark fixture set it on the mock
    (config.variant_data_proxy = None).

Separately: should the benchmark job collect the whole directory, so the non-timed tests in
it are at least executed? Today a benchmark that stops working is invisible until someone
runs the task locally.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pkg: sphinx-needsThe sphinx-needs distribution (packages/sphinx-needs): its code, tests and docs

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions