Skip to content

Isolate malformed LLM batch responses #265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
koriyoshi2041 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Isolate malformed LLM batch responses #265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 28 additions & 4 deletions src/skillspector/llm_analyzer_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
from typing import Literal

from langchain_core.messages import BaseMessage
from pydantic import BaseModel, Field, field_validator
from pydantic import BaseModel, Field, ValidationError, field_validator

from skillspector.llm_utils import get_chat_model
from skillspector.logging_config import get_logger
Expand Down Expand Up @@ -386,11 +386,35 @@ def run_batches(
len(batch.findings),
)
if self._structured_llm:
response = self._structured_llm.invoke(prompt)
try:

@rng1995 rng1995 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainability: this is the first of three near-identical try/except blocks in the loop. Extracting the policy into a small helper (skip on ValidationError/other Exceptions, propagate ValueError/NotImplementedError) would keep it in one place and let arun_batches share it, preventing the sync/async drift that left the async path unfixed. Also, the run_batches docstring should document the new failure-isolation semantics the way arun_batches's docstring does.

response = self._structured_llm.invoke(prompt)
except ValidationError as exc:

@rng1995 rng1995 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This carve-out is correct here, but the same fix is missing in arun_batches: pydantic ValidationError subclasses ValueError, so isinstance(result, (ValueError, NotImplementedError)) at line 441 re-raises it, and all three async callers (semantic_quality_policy, semantic_developer_intent, meta_analyzer) re-raise ValueError at node level — the whole-scan crash from #250 is still reproducible through those stages (verified against this branch with the same We{"findings":[]} payload). Please add an isinstance(result, ValidationError) check before the ValueError re-raise in arun_batches, plus an async regression test.

logger.warning("LLM batch failed for %s: %s", batch.file_label, exc)

@rng1995 rng1995 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping the batch with only a log warning gives callers no signal. Previously a ValidationError reached semantic_security_discovery's node-level handler, which recorded llm_call_record(ok=False, error="malformed LLM response: ...") and fed the report's degradation notice; that handler is now dead code, so a scan where every batch fails to parse still reports the LLM stage as fully OK. Consider having the node compare submitted vs returned batches (as meta_analyzer does) and record a degraded call, so partial results stay annotated in the report.

continue
except (ValueError, NotImplementedError):
raise
except Exception as exc:
logger.warning("LLM batch failed for %s: %s", batch.file_label, exc)
continue
else:
response = _message_text(self._llm.invoke(prompt))
try:
response = _message_text(self._llm.invoke(prompt))

@rng1995 rng1995 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistency: this raw-mode branch has no ValidationError carve-out, so a ValidationError raised here (or in a subclass's raw-mode pipeline) propagates via the ValueError re-raise, while the structured branch above skips it. If intentional, add a comment; otherwise align the two branches.

except (ValueError, NotImplementedError):
raise
except Exception as exc:
logger.warning("LLM batch failed for %s: %s", batch.file_label, exc)
continue
logger.debug("LLM response for %s", batch.file_label)
parsed = self.parse_response(response, batch)
try:
parsed = self.parse_response(response, batch)
except ValidationError as exc:
logger.warning("LLM batch parse failed for %s: %s", batch.file_label, exc)
continue
except (ValueError, NotImplementedError):
raise
except Exception as exc:
logger.warning("LLM batch parse failed for %s: %s", batch.file_label, exc)
continue
results.append((batch, parsed))
return results

Expand Down
44 changes: 44 additions & 0 deletions tests/nodes/test_llm_analyzer_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,50 @@ async def test_arun_batches_uses_message_text_for_content_blocks(self) -> None:
assert results[0][1] == ["async chunk"]


# ---------------------------------------------------------------------------
# LLMAnalyzerBase.run_batches (sync execution)
# ---------------------------------------------------------------------------


class TestRunBatches:
MODEL = "nvidia/openai/gpt-oss-120b"

@patch(MOCK_PATCH_TARGET, _mock_get_chat_model)
def test_malformed_structured_batch_does_not_abort_the_others(self) -> None:

@rng1995 rng1995 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good regression test for the sync path. Please add the async counterpart: an arun_batches test where ainvoke raises this same ValidationError and the other batches survive — it currently fails on this branch because arun_batches re-raises ValidationError via the (ValueError, NotImplementedError) isinstance check. Coverage for the parse_response leg and raw-string mode of the new sync handling would also be worthwhile.

"""A malformed structured response costs only its own batch."""

def _invoke(prompt: str) -> LLMAnalysisResult:
if "b.py" in prompt:
return LLMAnalysisResult.model_validate({"findings": 'We{"findings":[]}'})
return LLMAnalysisResult(
findings=[
LLMFinding(rule_id="T-1", message="hit", severity="LOW", start_line=1),
]
)

analyzer = LLMAnalyzerBase(base_prompt="test", model=self.MODEL)
analyzer._structured_llm.invoke.side_effect = _invoke

batches = [
Batch(file_path="a.py", content="code a"),
Batch(file_path="b.py", content="code b"),
Batch(file_path="c.py", content="code c"),
]
results = analyzer.run_batches(batches)

assert {batch.file_path for batch, _ in results} == {"a.py", "c.py"}
assert [items[0].rule_id for _, items in results] == ["T-1", "T-1"]

@patch(MOCK_PATCH_TARGET, _mock_get_chat_model)
def test_value_error_still_propagates(self) -> None:
"""ValueError signals misconfiguration, not a malformed model response."""
analyzer = LLMAnalyzerBase(base_prompt="test", model=self.MODEL)
analyzer._structured_llm.invoke.side_effect = ValueError("no API key")

with pytest.raises(ValueError, match="no API key"):
analyzer.run_batches([Batch(file_path="a.py", content="code")])


# ---------------------------------------------------------------------------
# LLMAnalyzerBase.arun_batches (async parallel execution)
# ---------------------------------------------------------------------------
Expand Down