In-situ muon efficiency method for the Z dilepton and W analyses#709
Draft
davidwalter2 wants to merge 18 commits into
Draft
In-situ muon efficiency method for the Z dilepton and W analyses#709davidwalter2 wants to merge 18 commits into
davidwalter2 wants to merge 18 commits into
Conversation
…0216 Snapshot of uncommitted work (in-situ ID/HLT/Iso efficiency implementation and pre-existing local edits) so the working tree is clean for merging origin/260216_totalPhaseSpace. Co-Authored-By: Claude Opus 4.7 <[email protected]>
…216_totalPhaseSpace # Conflicts: # scripts/corrections/corrs_by_helicity/make_pdf_gen_hists.py # scripts/corrections/make_theory_corr.py # scripts/corrections/plot_theory_corr.py # wremnants/production/datasets/datasetDict_13TeVGen.py # wremnants/utilities/styles/styles.py
- Make dilepton trigger selection the only mode in mz_dilepton.py (drop
--useDileptonTriggerSelection, --muonIsolation; remove the wlike branch).
- Add reco/tracking effSyst altBkg per-category emission so setupRabbit
--muonInsituEfficiency finds nominal_effSystTnP_altBkg_{reco,tracking}.
- Fix effStat crash for reco/tracking: replace global smooth3D with the
per-step is3D flag in muon_efficiencies_smooth.py, so base steps build
and fill a rank-5 stat hist consistent with the non-utDep helper
(resolves "no. of axes != no. of indices").
- Drop unused datasetGroup arg from select_good_muons and the four
histmaker callers.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
…iles
Histmaker (mz_dilepton.py):
- effMCprobe filling fills BOTH tag-and-probe legs of 2HLT events (each routed
by its own isolation, at its own eta/pt/charge/uT) so the MC efficiencies use
the exact dilepton tag-and-probe counting (fixes the ~8% trigger bias from
one-random-probe-per-event); gated behind --makeInsituEffMC.
- Per-(eta,pt,[charge]) uT-quantile fit axis for failIso/failHLT via
make_quantile_helper (--makeUTQuantileHists pre-step + --utQuantileFile);
clamp recoUT into the flow-less helper axis range to avoid an out-of-bounds
crash on high-pT-Z tails. gen-uT (helper) vs reco-uT (fit axis) wiring.
effMC producer (make_insitu_effMC.py):
- One pkl per process (--process nargs); diagnostic plots overlay all processes
with stat error bars, per-step subfolders + per-cell subsubfolders, adaptive
y-range (driven by the reference process), dashed unity line.
- Optional tag-and-probe overlay (--tagAndProbeDir): EffMC2D/EffData2D as data
points with stat + bin-width error bars on the per-cell pT curves.
- Provenance fix: nest the input file's metadata so the .log keeps this
script's own command/git hash.
Helper (muon_efficiencies_insitu.{hpp,py}, systematics.py, rabbit_helpers.py):
- 2D (pT,uT) Chebyshev for HLT/Iso (charge-dependent HLT), matching
SMP-23-002; 2112 unconstrained nuisances. Shared base class with a
single-leg variant for the W single-muon topology.
New scripts/plotting/plot_uT_quantile_distributions.py: per-cell uT
distributions used to build the quantiles, with the quantile boundaries
overlaid.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Helper (muon_efficiencies_insitu.hpp/.py): - Reparameterise the data efficiency in logit space: eps = e*e^s/(e*e^s + 1-e) in (0,1), f_pass = e^s/D, f_fail = 1/D with D = e*e^s + (1-e). Both factors and their gradients (1-eps pass, -eps fail) are bounded, removing the (1-SF*e)->0 fail-factor singularity. Pass legs now use effMC. Dropped the SF*e clamp and the now-dead exponent/W guards. - effMC_max 0.999 -> 0.9999 so genuine high-eff idip cells (peak ~0.99976, 13% above 0.999) are no longer clamped, while e>=1 fluctuations still are. Histmaker (mz_dilepton.py): - Fold the iterative central reweight W(theta_central) into exp_weight before define_theory_weights_and_corrs, so the pdf/scetlib/EW weight tensors inherit it (previously only nominal+qcdScale were reweighted -> distorted syst/nominal and an inflated prefit band). Removed the late per-channel Redefine. - Upper pt cut 60 -> 70 GeV (binning.py z_dilepton default 44/26/70); effMC and fit axes reuse the variable axis_pt_failID; failHLT uT quantiles 7 -> 10. - Guard: --makeInsituEffMC and --insituSFFile are mutually exclusive. Diagnostics: - styles.py: functional labels for the 2112 in-situ Chebyshev coefficients. - New plot_insitu_effSF.py: per-cell SF +/- sigma_SF bands (full covariance, logit Jacobian) with external T&P overlay. - New make_insitu_effSF.py: iteration producer (accumulate theta_central from the fit pulls, convergence report). Co-Authored-By: Claude Opus 4.8 <[email protected]>
The in-situ efficiency work had reduced muon_efficiencies_smooth.hpp to
return only reco+tracking, deleting the full tag-and-probe helper family
(5-step scale_factor_array with pass/fail iso+trigger logic, the
muon_efficiency_smooth3D_helper* uT-dependent family, and the _stat_iso*
families). Restore those under their original names, and keep the reduced
reco+tracking helpers (used by the in-situ path) renamed to
muon_efficiency_recotrack_helper{_base,,_syst} so both families coexist.
make_muon_efficiency_helpers_smooth now dispatches on the requested eff
types: reco+tracking only -> recotrack helpers (Sizes<2> syst tensor);
otherwise the full tag-and-probe helpers. The full-TnP defaults were
already intact, so the in-situ path is unchanged (it just uses the
cleanly-named recotrack helpers).
Both families compile and instantiate against the real SF file. The
end-to-end tag-and-probe wiring in the histmakers (mz_dilepton event-loop
fork, systematics.py iso-aware syst columns, W-like define_trigger_muons)
is left for a dedicated follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Bring the W single-muon in-situ efficiency path up to parity with the
dilepton histmaker:
- effMC generation (--makeInsituEffMC): emit the four
effMCprobe_{nominal,failIso,failHLT,failID} probe spectra
(eta, pt, charge, genUT) from the prompt gen-matched veto muon, then
return early (before select_good_muons, so failID probes are kept).
Categories are defined on the single probe via its passID
(goodMuons mask, no event filter) / passHLT (trigger match) / passIso.
Consumed by scripts/corrections/make_insitu_effMC.py.
- iteration support (--insituSFFile): build and apply the single-leg
central reweight W(theta_central) into nominal_weight, and fix the
helper construction to the current 3-tuple return
(helper, central_helper, labels) with central_sf_file.
- mutual-exclusion guard for --makeInsituEffMC / --insituSFFile.
Validated end-to-end on 2016PostVFP Wmunu MC: histmaker -> effMCprobe_*
(all 4 categories populated) -> make_insitu_effMC.py -> insitu_effMC pkl
-> single-leg helper builds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Align the in-situ probe denominator with the T&P framework (Steve.py
BasicProbe_Muons): insituProbeMuons = good global muon WITHOUT the
looseId/dxybs veto cuts (those belong to the IDIP numerator;
Muon_correctedCharge != -99 kept as a technical exception). The event
count filters move onto this collection in both histmakers so probes
failing looseId/dxybs survive to the failID category. This closes the
effMC_idip discrepancy vs the T&P MC for the Z (edge bin 0.9965 vs
T&P ~0.995, previously ~1.000 from the veto-conditioned denominator).
Add a tag-unbiased Z effMC alongside the tag-based one: each charge leg
is an independent gen-matched probe (no tag, no OS pairing, no mass
window, no event HLT filter; branch forks before the HLT filter), emitted
as effMCprobeUnbiased_{cat}_{minus,plus}. This is the Z analog of the W
gen-based effMC and isolates the genuine process dependence of the
efficiencies from the tag-conditioning of the T&P denominators
(correlated two-leg failures; established with standalone diagnostics).
make_insitu_effMC.py: --unbiased flag (charge-leg sum, with fallback to
effMCprobe_* for inputs without the unbiased spectra, e.g. the single-muon
histmaker which is tag-unbiased by construction), and per-entry variants
in --process ('Zmumu:unbiased') so tag-based and unbiased curves can be
overlaid in one invocation (dashed linestyle for the unbiased variant).
Known caveat (under investigation): the unbiased trigger effMC comes out
biased high (~0.89 vs ~0.81 truth at the eta edge) with a distorted uT
shape; the per-leg counting logic was validated standalone, the prime
suspect is the flow-less genUT axis [-30,100] clipping uT tails that are
larger without the Z-candidate selection and correlated with HLT-iso
failures. idip is unaffected (uT-independent recombination).
Also includes further in-situ developments in the dilepton histmaker
(insituMode restructuring with restored tag-and-probe selection path),
muon_efficiencies_insitu, systematics, and the in-situ SF plotting.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fix: hlt_string() returns "HLT_IsoTkMu24 || HLT_IsoMu24"; embedding it
unparenthesized in "{hlt} && hasTriggerMatch(...)" parses as
"IsoTkMu24 || (IsoMu24 && match)" since && binds tighter than ||, so any
probe whose PARTNER fired IsoTkMu24 counted as triggering regardless of
its own matching. This inflated the tag-unbiased Z trigger effMC by ~8%
(0.89 vs 0.81 truth at the eta edge) with a distorted uT shape, gutting
the failHLT category; idip was untouched (the flag only moves probes
between nominal/failIso/failHLT). The single-muon W paths were protected
by construction (only the probe itself can set the event bit), which is
why the W effMC agreed with T&P while the unbiased Z did not. Fixed in
three sites: the mz unbiased branch, the mw effMC branch, and the
pre-existing mw --makeMCefficiency path (same hazard for events with a
second muon, e.g. Zmumu MC). The main analysis selections are unaffected:
there the HLT bits are a standalone event filter and the per-muon
matching never embeds them. Validated on a 5-file smoke: unbiased
eff_trig 0.8897 -> 0.8222, now consistent with tag-based (0.83) and the
standalone truth (0.811); iso routing also recovers.
Clamp uT into the flow-less genUT axis window at fill time
(insitu_ut_clamp_expr, applied at all four effMC fill sites): out-of-
window probes (~1%, fail-enriched 2-3x in trigger/iso) previously
dropped from the effMC spectra while the C++ helper clamps uT at
evaluation time and applies the edge-bin effMC to them. The edge bins
are now measured on exactly the population they are applied to
(<=1e-3 on integrated trigger/iso effMC, none on idip). Validated:
+1.4% probes recovered into the uT edge bins, eff_trig -1e-3, idip
bit-identical.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
…t 26-90 make_insitu_effMC.py: - --smooth: fit each (eta[, charge]) cell with the same logit-Chebyshev form, clamped x-tilde windows and pt-major coefficient layout as the in-situ SF parameterisation (orders via --smoothCoeffPt/Ut, defaults = SF orders; full-stat shapes want ~6x5). Variance-weighted least squares with a boundary-aware sigma floor 1/(n_eff+2) so eff>=1 zero-variance bins carry finite weight. The pkl effMC becomes the smoothed one (drop-in for the helper), with effMC_raw and the fit metadata/coefficients alongside; per-cell plots overlay the smooth curves (den-weighted uT average for the uT-integrated ones). - Correlation factor C = eff(T&P-like)/eff(w/o tag) for every tag/unbiased pair in --process: 1D projections, 2D (eta,pt) maps per charge, (pt,uT) maps eta-integrated and per eta slice, and per-cell C(pT) plots in the native parameterisation cells; with --smooth also derived from the smoothed efficiencies (overlay lines on 1D, parallel _smooth 2D maps). C cancels in the pass weight Ce'/(Ce) but survives in the fail weight (1-Ce')/(1-Ce) with leverage Ce/(1-Ce); using the tag-based effMC in the Z helper implements C from MC to first order -- these plots quantify it. - Labels: '(w/o tag)' for tag-unbiased entries incl. the single-muon histmaker ones (detected via source mode), dashed = w/o tag uniformly, w/o-tag entries grouped first in the legends. - --plotPtRange (default 26-65): displayed pt range of every plot with a pt axis, so productions with different upper pt cuts compare directly; y-ranges computed from the visible window only. effMC probe-pt range extension (both histmakers): - insitu_effMC_pt_edges(): fixed grid [26..50, 65, 90] decoupled from the per-analysis --pt cut; make_insitu_effMC_axes uses it (pt_max dropped). - effMC probe selections extended to the axis range with the goodMuons mask given the same window, so passID stays a pure ID decision up to 90; the mz tag-based failID fill is masked at the analysis cut where the fit's goodMuons window would otherwise misclassify ID-passing probes (all four tag-based categories then bounded consistently; with --pt ... 90 the mask is a no-op and the tag-based spectra populate the full grid). - Chebyshev pt window insitu_pt_range -> (26, 90), matching the effMC grid. NB this redefines the coefficient nuisances: theta_central pkls, fit results and effMC files from the (26,70) era are not compatible and need regeneration (the smoothed-pkl metadata records the window). Co-Authored-By: Claude Opus 4.8 <[email protected]>
…sed) The two-leg in-situ weight model previously used one effMC set for both legs, implicitly putting the tag-conditioned effMC also on the tag leg. With C = e_cond/e_incl the pair probability factorises as P = C * e_probe * e_tag, so the conditioning belongs on exactly one leg: the probe uses the tag-conditioned effMC (matching the tag-conditioned fit categories) and the tag the tag-unbiased one, counting the correlation once. The previous single-set behaviour mildly mis-linearised the tag-leg factors (a (C-1)-sized effect at high |eta|). - C++ helper base carries two effMC triplets (probe + tag); the tag-leg calls in compute<WithTag> and central_weight<WithTag> look up the tag triplet. theta and all derivatives stay shared (the SF parameterisation is per-muon universal; only the linearisation effMC differs per leg). Single-leg helpers unaffected (probe triplet passed twice). - make_muon_insitu_efficiency_helper / setup_muon_insitu_helpers gain tag_effMC_file (None -> probe file for both legs, bit-identical to the previous behaviour, verified by direct functor evaluation). - mz_dilepton: --insituEffMCFileTag feeds the tag leg for all MC (typically insitu_effMC_Zmumu_unbiased from the smoothed producer); the probe leg keeps --insituEffMCFile. Smoothed efficiencies enter by pointing both at pkls produced with make_insitu_effMC.py --smooth. - mw unchanged: single leg, the (tag-unbiased by construction) W effMC. Validated: JIT closure for all helper variants; exact regression for tag file = probe file; small finite tag-leg-only weight shifts with the unbiased tag effMC and a toy theta_central; full smoke of the dilepton fit mode with both files producing the in-situ variation histograms for signal and background datasets. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replace the logit parameterization of the in-situ ID/HLT/Iso scale factors with a linear one. The passing-leg weight is now the Chebyshev polynomial directly, f_pass = 1 + P (P unrestricted, may exit [0,1]) f_fail = (1 - eMC*(1+P)) / (1 - eMC) so the fitted coefficients are a pure data/MC ratio with NO dependence on the MC efficiency: eMC enters only the fail factor. Each analysis can therefore use its own effMC via a single --insituEffMCFile (W effMC for mw, Z tag-and-probe effMC for mz_dilepton). The dilepton tag leg always passes, so its weight (1+P) is MC-free -- the probe/tag effMC split and --insituEffMCFileTag added in 38971b4 are reverted to one effMC triplet. The logit form bounded the implied data efficiency in (0,1) at the cost of an eMC-dependent f_pass; the linear form trades that back: where the data efficiency eMC*(1+P) reaches 1 (or 1+P <= 0) the helper now THROWS (no clamp) -- a deliberate probe of whether the data keep P inside the physical region. At central |eta| (idip eMC ~ 0.99) even a small positive P triggers it. delta (0.01) may need retuning since the fail derivative ~ eMC/(1-eMC) is no longer bounded. - hpp: rewrite eval_leg (returns the polynomial value, no exp/clamp), accumulate and central_factor for the linear factors and their gradients; add throw_unphysical; drop the tag effMC triplet and the bool tag selector; rewrite the doc comment. - muon_efficiencies_insitu.py: drop tag_effMC_file; instantiate with the three probe hists only. - mz_dilepton: drop --insituEffMCFileTag. mw unchanged (single leg, W effMC). Validated: nominal preserved at theta=0; f_pass/f_fail and their gradients reproduced to machine precision; throw fires when eMC*(1+P) >= 1; passing weight identical across Z and W effMC while the failing weight tracks each; full mz_dilepton (4 category tensors per Z process) and mw single-leg in-situ smoke runs complete. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Bring the in-situ muon efficiency branch up to date with the total
phase space line (74 commits): postfit PDF grids + theory uncertainties
applied via correction hists by default (--{pdf,as,minnlo}UncFromWeights
opt-out), low/high-mass DY harmonization, rabbit bump to 73f346aa, and
related setupRabbit/datagroups/histselections updates.
Conflicts (both trivial, resolved to the incoming formatting):
- datagroups.py: blank-line differences in the OnesSelector none-mode path
- histselections.py: trailing period in a closure-test comment
In-situ work (linear f_pass=1+P helper, --insituEffMCFile, single effMC
triplet) auto-merged cleanly and verified present alongside the incoming
PDF/theory features.
For the uT-dependent steps (trigger, iso), add two diagnostic plot families to make_effMC_plots, alongside the existing eff(pT) per-cell and per_cell_ut plots: - per_cell_pt/eta<i>/effMC_<step>_ut_eta<i>_pt<j>[_q]: the transpose of per_cell_ut, eff(uT) at a fixed (eta, pT) cell, with the smoothed curve overlaid at the pT bin center. - per_cell_2d/effMC_<step>_2d_eta<i>[_q]: a 2D eff(pT, uT) heatmap per (eta, charge) cell for the reference process; with --smooth it is a raw | smoothed | pull=(smooth-raw)/sigma triptych to localise mis-description. Add a _smooth_grid_cell() helper that evaluates the fitted surface on an arbitrary (pT, uT) grid, reused by both. idip (no uT) is skipped. These make the high-uT smoothing/SF behaviour directly visible. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Update the post-fit SF diagnostic to the linear parameterization (the passing-leg SF is the Chebyshev polynomial directly, SF = 1 + P, MC independent), replacing the obsolete logit form SF = exp(s)/D = eps/e: - sf_band: SF = 1 + P with sigma_SF = sqrt(B^T Cov B) (SF is linear in theta, so a plain Jacobian; no (1-eps) factor, no effMC). Drop the --effMCFile argument and the eps/effMC machinery entirely. - Add --sfFile: plot the SF curves straight from an accumulated theta_central pkl (insitu_effSF*.pkl.lz4), without a fit result and without a covariance band, to inspect a produced SF (e.g. high-uT singularities). Mutually exclusive with -i. - Add --ptPlotRange: zoom the displayed pT axis independently of --ptRange, which must stay the production Chebyshev WINDOW (26,90); setting the window to the fit range mis-normalises the SF. - Suppress the band/legend when plotting without a covariance. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two related changes to the in-situ muon efficiency parameterisation. pt window 26-90 -> 26-65. The 90 GeV upper edge only existed to give the effMC smoothing high-pt leverage; smoothing is no longer used, so the Chebyshev window is cut back to the analysis range. insitu_effMC_pt_edges() now returns exactly insitu_pt_edges(65), so the effMC probe pt axis and the dilepton failID fit axis are identical and the effMC and uT-quantile pre-steps can share a single histmaker run. Coefficient count is unchanged at 2112, but the x~ transform is redefined: earlier theta_central pkls and fit results are not compatible. Basis orthogonalisation (--insituBasisFile, optional). Chebyshev is orthogonal under 1/sqrt(1-x^2), not under the probe density, so the coefficients came out strongly correlated (max |rho| 0.93-0.99 in every (step, eta) block; block condition numbers up to 1e5), which is what makes the fit need repeated refits. Changing polynomial family does not help (Legendre is no better) and narrowing the uT window barely does. Instead make_insitu_basis.py builds per-block transforms A = rownorm(L^-1) from the Cholesky of the Gram matrix of the raw basis under the fit's own weight w = n_pass + n_fail*(eMC/(1-eMC))^2, taken from the effMCprobe_* spectra. The helper applies b' = A b in eval_leg; everything downstream follows. The row normalisation matters: bare L^-1 has large rows along the weakly-constrained directions, where the fail-category probes live, and blew the stored exp(delta*grad) up to ~1e26. Rescaling each row to max_x |b'_j(x)| = 1 over the clamped window restores exactly the raw Chebyshev response magnitudes while leaving the Gram diagonal, since a diagonal rescaling cannot reintroduce correlations. This is a pure reparameterisation - the fitted scale factors and their uncertainty band are unchanged, only the conditioning. Verified: grad' = A grad to 8.5e-14 at functor level, nominal templates agree to 1e-16 between raw and orthogonalised runs, A invertible in all 240 blocks, and the Gram correlation condition number is exactly 1.000 everywhere. The transform must be frozen and shared across analyses that correlate the same coefficient nuisances, hence a standalone input file rather than something each histmaker derives; the loader validates it against the coefficient counts and both x~ windows. Omitting the flag gives identity, i.e. the previous behaviour exactly. make_insitu_effSF.py needs no change (its theta accumulation is label-keyed and basis-agnostic), but plot_insitu_effSF.py evaluates the polynomial itself and gets the same flag - it must be run with the same basis file as the histmaker. Also adds plot_insitu_correlations.py for the aggregate correlation view, which rabbit_plot_cov.py --params cannot draw above ~50 parameters. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — opened for visibility while the scope is still being split. Not ready to merge as-is, see Scope caveats below.
What this adds
An in-situ determination of the muon ID / HLT / Iso data-MC scale factors, as an alternative to the external tag-and-probe SFs. The three efficiency steps are floated as Chebyshev polynomials in (pT, uT), decorrelated per probe-eta bin, as unconstrained nuisances (not NOIs), so the scale factors are determined by the fit itself rather than taken from an external measurement.
The Z dilepton analysis is split into four categories —
nominal,failIso,failHLT,failID— which become four fit channels sharing the same coefficient nuisances, and that cross-category constraint is what determines the efficiencies. Reco and tracking SFs stay external.The passing-leg scale factor is the polynomial directly,
f_pass = 1 + P, which makes it MC-independent; the MC efficiency enters only the failing-leg factorf_fail = (1 - eMC(1+P))/(1 - eMC). Each analysis therefore supplies its own effMC. The method is implemented for both the dilepton (two-leg tag-and-probe) and the W single-muon (single-leg) topologies through a shared C++ base.The exact likelihood is linearised around a central point and iterated:
make_insitu_effSF.pyaccumulatestheta_new = theta_prev + delta * n_hat, fed back via--insituSFFile.Workflow
mz_dilepton.py --makeInsituEffMC --makeUTQuantileHists— probe spectra and uT quantile inputsscripts/corrections/make_insitu_effMC.py— per-step MC efficienciesscripts/corrections/make_insitu_basis.py— basis orthogonalisation (optional, see below)mz_dilepton.py --insituEffMCFile ... [--insituBasisFile ...]— templates and coefficient response tensorssetupRabbit.py --muonInsituEfficiencywith the four--baseNamesmake_insitu_effSF.pyto produce the next iteration's central coefficientsBasis orthogonalisation
Chebyshev polynomials are orthogonal under the weight
1/sqrt(1-x^2), which is nothing like the probe density, so the coefficients came out strongly correlated —max |rho|of 0.93–0.99 in every (step, eta) block, with block condition numbers up to 1e5. That is what made the fit need repeated refits to converge.Measurements showed changing polynomial family does not help (Legendre condition number 5976 vs Chebyshev 7723) and narrowing the uT window barely does.
make_insitu_basis.pyinstead builds per-block transformsA = rownorm(L^-1)from the Cholesky of the Gram matrix of the raw basis under the fit's own weight,w = n_pass + n_fail*(eMC/(1-eMC))^2. This drives the Gram correlation condition number to exactly 1.000 in all 240 blocks.The row normalisation is load-bearing: bare
L^-1has large rows along the weakly-constrained directions where the fail-category probes live, which blew the storedexp(delta*grad)up to ~1e26. Rescaling each row tomax_x |b'_j(x)| = 1restores exactly the raw Chebyshev response magnitudes, and since a diagonal rescaling cannot reintroduce correlations the decorrelation is untouched.This is a pure reparameterisation — the fitted scale factors and their uncertainty band are unchanged, only the conditioning. Verified:
grad' = A gradto 8.5e-14 at functor level, nominal templates agree to 1e-16 between raw and orthogonalised runs, andAis invertible in all blocks. Omitting--insituBasisFilegives the identity, i.e. the raw Chebyshev basis exactly.Scope caveats
260216_totalPhaseSpacebranch (plot_theory_corr.py,make_pixel_corrections.py,setup.sh,.gitignore). Those are not part of this feature and should be split out or land separately.--insituEffMCFilethe histmakers follow the existing external tag-and-probe path unchanged.🤖 Generated with Claude Code