Skip to content

fix(preprocess): parse shared exchange ticker lists#129

Draft
pengpengyi92 wants to merge 2 commits into
LLMQuant:masterfrom
pengpengyi92:fix/issue-108-exchange-symbol-lists
Draft

fix(preprocess): parse shared exchange ticker lists#129
pengpengyi92 wants to merge 2 commits into
LLMQuant:masterfrom
pengpengyi92:fix/issue-108-exchange-symbol-lists

Conversation

@pengpengyi92

Copy link
Copy Markdown

Summary

  • parse parenthesized comma-separated ticker lists that share a supported exchange prefix
  • stop shared-prefix parsing at semicolons, conjunctions, and closing parentheses
  • define reconstructed EXCHANGE: SYMBOL raw values for additional list members
  • add deterministic regression coverage for the five public shapes from the issue, plus a focused example and design documentation

Related Issue

Closes #108

Verification

  • pytest -o addopts='' tests/preprocess/test_news.py -q - 20 passed, 8 subtests passed
  • ruff format --check . - passed
  • ruff check . - passed
  • basedpyright - passed with 0 errors
  • lint-imports - 7 contracts kept, 0 broken
  • python examples/preprocess/02_news_ticker_lists.py - emitted both EVEX and EVEXW
  • Full pytest --cov on Windows reached 84.05% coverage with 322 passed and 13 pre-existing platform failures involving SQLite temporary-file locks and LiteParse artifact paths; this draft relies on the repository's Linux CI for the required deterministic verification.

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • bash scripts/verify.sh passes locally; Windows-specific existing failures are disclosed above and Linux CI is pending.
  • No live-network component smoke test applies because this change is deterministic parsing only.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes.

pengpengyi92 and others added 2 commits July 21, 2026 23:35
The files on this branch were committed with CRLF line terminators, which marked every line as changed and produced spurious merge conflicts with master. Convert them back to LF so the diff reflects only the real behavior change from LLMQuant#108.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

@keli-wen keli-wen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. Thanks for taking on #108 — the intent is right, but the shared-prefix extension as written is a net regression, and I don't think this change should carry a new example or a design-doc rewrite. Details below.

Blocking: the shared-prefix scan over-captures ordinary prose

The continuation loop keys off match.end() and then searches for the next ) anywhere in the text (scan_text.find(")", match.end())). When the main regex already matched a balanced (EXCHANGE: SYMBOL) — it consumed its own closing paren — the loop still runs and scans past that paren into unrelated following text. Because _SHARED_EXCHANGE_SYMBOL_RE is compiled with re.IGNORECASE, an ordinary lowercase word right after a (EXCHANGE: SYMBOL), is captured as a ticker.

This flows straight into preprocess_news_document (ticker_hints, news.py:330), so it pollutes real output. On current master the input below yields one hint; on this branch it yields a bogus second one:

# (symbol, exchange, raw) projected from each returned NewsTickerHint
>>> extract_exchange_ticker_hints("Shares of (NYSE: IBM), and (NASDAQ: AAPL) rose today.")
('IBM',  'NYSE',   '(NYSE: IBM)')
('AND',  'NYSE',   'NYSE: AND')      # <- "and" captured as ticker "AND"
('AAPL', 'NASDAQ', '(NASDAQ: AAPL)')

The trigger — a balanced (EXCH: SYM), followed by any later ) — is extremely common in PR-wire prose, so this regresses far more documents than the ~1 warrant of recall that #108 estimates it recovers. The added tests only assert the five happy-path shapes from the issue and never exercise this case, so CI stays green while the regression ships.

Related smell: raw is meant to be the literal matched substring (provenance), but continuation members set raw=f"{exchange}: {symbol}" — a string that never appears in the source (NYSE: EVEXW is reconstructed) — while the first member keeps a dangling (NYSE: EVEX. The tests encode both, which locks that broken raw contract in.

This change should not add an example

extract_exchange_ticker_hints already exists and is already demonstrated end-to-end by examples/preprocess/01_news_pr_wire.py. Extending an existing operation's parsing coverage is a fix, not a new public operation, so it should ship regression tests rather than a second example file. Please drop examples/preprocess/02_news_ticker_lists.py. We'll also tighten the contributor guidance so the "ships with an example" expectation is scoped to new public operations, not edge-case extensions.

The contexts/ doc change is questionable here too

Whether a parsing edge-case should rewrite contexts/design/flow/news.md is debatable — unless this actually changes a documented contract, the design doc probably shouldn't move. It is also the only real merge conflict with master right now (#130 rewrote that file), so dropping the doc change both narrows scope and clears the conflict.

Suggested path

Narrow this to what #108 is actually worth: keep the existing simple/balanced capture and add regression tests that pin the boundary — the first symbol of a shared-prefix list is captured, and (EXCH: SYM), ...) does not over-capture. If we would rather not carry the heuristic at all, deprecating ticker_hints is a reasonable alternative; happy to discuss which way you'd prefer.

Note: I pushed a small chore: normalize CRLF line endings to LF commit to this branch. The files were saved with CRLF, which made the diff look like a full-file rewrite; it now reads as ~+89/-4 against the merge base.

@keli-wen keli-wen added type: bug Existing behavior violates its contract or documented expectation area: preprocess Deterministic acquisition, parsing, cleaning, and source handling labels Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: preprocess Deterministic acquisition, parsing, cleaning, and source handling type: bug Existing behavior violates its contract or documented expectation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: exchange-qualified multi-symbol lists are only partially captured

2 participants