[BUG] ctrld v1.5.3: DoQ Upstream Response Rewriting Bypasses Failover and Pollutes Cache

[N0zoM1z0]

Summary

This report covers the DoQ upstream resolver path in ctrld v1.5.3.

After reading a DoQ response from the upstream, ctrld calls SetReply(request)
on the already formed upstream response. That rewrites response fields that must
be preserved, including the RCODE and Question section.

The same root cause has two observable effects:

1. A DoQ upstream SERVFAIL response is rewritten to NOERROR before
   failover_rcodes is evaluated, so a configured fallback upstream is not
   queried.
2. A wrong-question DoQ response can be rewritten into the victim request
   context and cached under the victim query key while the answer RRset still
   belongs to another name.

Verified version:

• Product: ctrld
• Release: v1.5.3
• Source commit: 5bf26da585d92d2352d26adebfcd80de5a152823
• Affected component: DoQ upstream resolution

Bundled evidence:

• failover summary:
  attachments/results/shipped/failover/summary.md
• failover test log:
  attachments/results/shipped/failover/go-test.log
• cache summary:
  attachments/results/shipped/cache/summary.md
• cache test log:
  attachments/results/shipped/cache/go-test.log

Root Cause Analysis

The DoQ resolver path normalizes an upstream response by calling
SetReply(request) after the response has already been unpacked.

SetReply is useful when constructing a new DNS response from a request. It is
not safe as a general normalization step for an upstream response because it
overwrites response state derived from the request. On this path, that changes
the meaning of the upstream response before downstream policy and cache logic
consume it.

The important fields are:

• Rcode
• Question
• reply header fields derived from the request

Only the client-facing transaction ID needs to be restored for the downstream
DNS response. The upstream RCODE, Question, Answer, Authority, and Additional
sections should remain the upstream response state until validation and policy
decisions have completed.

Impact 1: SERVFAIL Is Rewritten to NOERROR Before Failover

In the tested failover profile:

• the client asks ctrld for example.test. A
• the primary DoQ upstream returns SERVFAIL
• ctrld rewrites that response to NOERROR
• failover_rcodes = ["SERVFAIL"] no longer matches
• the fallback upstream is never queried

Observed result:

rewritten_rcode: NOERROR
answer_count: 0
fallback_hits: 0
failover_bypass_confirmed: true

This breaks the configured failover policy. Clients receive a misleading
successful empty response, and resolver routing decisions are made on rewritten
state rather than the actual upstream RCODE.

Impact 2: Wrong-Question Response Is Cached Under the Victim Key

In the tested cache profile:

• the client asks ctrld for victim.example. A
• the DoQ upstream returns a response for attacker.example. A
• ctrld rewrites the response Question to victim.example. A
• the Answer section still contains attacker.example. A 203.0.113.66
• the shared cache stores the response under the victim.example. A key
• the second victim.example. A query is served from cache

Observed result:

first_question: victim.example.
first_answer: attacker.example. 300 IN A 203.0.113.66
second_question: victim.example.
second_answer: attacker.example. 299 IN A 203.0.113.66
upstream_hits: 1
wrong_question_cache_pollution_confirmed: true

This is a cache-integrity issue. The cache entry for one DNS question can be
populated with an RRset owned by another DNS name. Later queries for the victim
key are answered from invalid cross-name cache state instead of a fresh upstream
answer for the victim name.

Reproduction Steps

The attachment directory is standalone:

• instructions: attachments/README.md
• script: attachments/scripts/run_repro.sh
• package-level test:
  attachments/pocs/doq_response_rewrite_repro_test.go

From attachments/:

bash scripts/run_repro.sh

If attachments/upstream does not exist, the script clones the tested ctrld
snapshot automatically. To use an existing checkout:

UPSTREAM_DIR=/path/to/ctrld bash scripts/run_repro.sh

The script temporarily copies the package-level test into cmd/cli, runs both
test cases against ctrld's internal resolver path, writes logs under
results/runs/<RUN_ID>/, and removes the temporary test file on exit.

Expected combined summary:

failover_bypass_confirmed: true
wrong_question_cache_pollution_confirmed: true

Recommended Fix

• Do not call SetReply on an already formed upstream DoQ response.
• Restore only the downstream DNS transaction ID that belongs to the client
  request.
• Preserve upstream Rcode, Question, Answer, Authority, and
  Additional sections for validation, failover, and cache decisions.
• Validate that the upstream response Question matches the original request
  QNAME, QTYPE, and QCLASS before returning or caching the response.
• Evaluate failover_rcodes against the original upstream response RCODE.
• Add DoQ regression tests for:
  • SERVFAIL / NXDOMAIN failover behavior
  • wrong-question responses
  • wrong-question responses with cache enabled

Attachments

attachments.zip

[cuonglm]

Thanks for reporting and details anaylysis.

This will be fixed in next release of ctrld.

[cuonglm]

@N0zoM1z0 This has been fixed with the latest ctrld dev version. You can try by running:

ctrld upgrade dev

[N0zoM1z0]

Hi @cuonglm ,

Thanks for the quick update and for addressing this so fast! I see the commits cover both the SetReply rewrite and the question mismatch logic. I'll update my environments to the dev version.

Thanks again for your excellent work on maintaining this project!