Issue #312: Bug: extract_json_from_response fails to strip bare ``` code fences (no 'json' tag)
Recommended: merge #313
fix: strip bare ``` code fences in extract_json_from_response
How it breaks
The path is the model's reasoning. Each box is a citation checked against the code.
What's wrong
extract_json_from_response only strips a leading fence when it is exactly "```json" (7 chars), but strips any trailing "```". A bare "```" opening fence is left in place, so the cleaned text starts with "```" and json.loads fails.Show full explanation
The three callers (pdf.py, github.py, evaluator.py) pass the result straight to json.loads. The report's claims are accurate. One difference: the current code is not the regex the report proposes, and the regex is only a suggested fix. Also, a "```JSON" fence, or a fence that follows leading text after the think tag is removed, would fail the same way.How to reproduce
- From the repo root run: python -I -c "from llm_utils import extract_json_from_response as e; import json; r=e('```\n{\"name\": \"test\", \"value\": 1}\n```'); print(repr(r)); json.loads(r)"
- The output is '```\n{"name": "test", "value": 1}\n', then json.loads raises JSONDecodeError: Expecting value: line 1 column 1 (char 0).
- Compare with the '```json\n...\n```' input, which parses correctly.
Evidence
Each citation was checked against the code at e0ece80.
The leading fence is only recognised with the literal 'json' tag, so a bare ``` opening fence is not handled.
The trailing fence is stripped with no tag check, so only the closing fence is removed from a bare-fenced response. This asymmetry leaves a dangling opening fence.
The cleaned text is parsed directly, so a leftover ``` raises JSONDecodeError in the scoring stage.
The same unsafe parse happens in the top-project selection stage.
The same unsafe parse happens in the section extraction stage.
Pull requests
Verdicts are judged from the diff, not run. Merge status is a real git check against e0ece80.
Replaces the literal ```json prefix check with a regex that accepts an optional json tag, so a bare ``` opening fence is stripped for all three callers.
Adds an elif startswith("```") branch that strips a bare opening fence, and then trims to the JSON object or array.
Swaps the prefix check for a ```json search plus brace matching, which strips bare fences for object responses, but the brace matcher would reduce the github.py project-selection list to its first object.
extract_json_from_response is untouched, but the new ensure_valid_json brace-slices the text in evaluator.py and pdf.py, which recovers bare-fenced objects there; github.py doesn't use it.
Slices from the first { to the last }, which removes bare fences for object responses, but it would turn the github.py list response into invalid JSON, including for ```json fences that worked before.
Adds a .strip() after removing the think block, which fixes a ```json fence behind leading whitespace, but a bare ``` opening fence is still not recognised.
Leaves extract_json_from_response alone, but evaluator.py now slices from the first { to the last } and retries, which recovers a bare-fenced object on that path only.
16 PRs that don't fix itShow
Changes github.py API fetching and retries and doesn't touch fence stripping.
Adds sectional snippets and retries in pdf.py and still uses the unchanged extract_json_from_response.
Only changes logging and print output and leaves the fence logic as it was.
The startswith("```json") check is unchanged, so a bare fence still leaves a dangling ``` and the function now returns None instead of fixing the parse.
Adds an open-source contribution pipeline in github.py and doesn't touch fence handling.
Adds a None check in evaluator.py, but extract_json_from_response never returns None in this code and the fence logic is unchanged.
Gemini structured output sidesteps fences on one provider, but extract_json_from_response keeps the same asymmetric fence logic and is still the fallback and the Ollama path.
The only llm_utils change is the parse_llm_response wrapper, which still calls the unchanged extract_json_from_response, so bare fences still fail.
Adds JD-augmented evaluation paths that call the same unchanged extract_json_from_response.
Validates bonus claims and project names after parsing and doesn't touch fence stripping.
Only changes the log level and stream routing, with no change to parsing.
Changes the author commit threshold in github.py and doesn't touch parsing.
Adds hallucination guards after parsing and leaves extract_json_from_response alone.
Adds a Bedrock transport, project resolution and sampling-parameter changes, and the fence stripping stays as it was.
Restores project metadata after parsing and doesn't change extract_json_from_response.
Changes log level, logger source and a message string only.
What this packet does not prove
- Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
- PR verdicts are judged from the diff, not run. No PR was applied or tested.
- PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.