Issue #68: Issue: Missing Error Handling for JSON Parsing in extract_json_from_response
No open PR fixes this and merges clean
1 fix it but don't merge clean, 7 fix it partly. See the board below.
How it breaks
The path is the model's reasoning. Each box is a citation checked against the code.
What's wrong
The report is partly right. extract_json_from_response (llm_utils.py:13) only strips <think> blocks and markdown fences. It never validates the result, so malformed JSON reaches the callers' json.loads() calls. The report overstates the impact.Show full explanation
Only evaluator.py lets the error escape: it logs and re-raises, so a malformed response fails the evaluation. pdf.py and github.py already catch JSONDecodeError and fall back (None, or the first 7 projects), so they do not crash. The report's quoted code also differs from the repo: it shows a "</<think>" typo that the repo does not have, since llm_utils.py:27 uses "</think>". Its line numbers are off too (pdf.py:113, github.py:403, evaluator.py:80). The cleanup is also fragile, because it only strips a fence at the very start or end and cannot handle prose around the JSON.How to reproduce
- In the repo root, run: python -I -c "from llm_utils import extract_json_from_response as e; print(repr(e('```json\n{bad json,}\n```')))"
- Observe it returns '\n{bad json,}\n' with no validation or error.
- Run json.loads on that string to get JSONDecodeError.
- In evaluator.py, mock provider.chat to return {'message': {'content': '```json\n{oops\n```'}} and call the evaluate method.
- The JSONDecodeError at evaluator.py:80 is logged and re-raised at line 87, so the evaluation fails.
- Run the same malformed input through the pdf.py section extraction or the github.py project selection. Both catch the error and fall back, so neither crashes.
Evidence
Each citation was checked against the code at e0ece80.
The function returns a plain string with no validation, error handling or None path.
It returns the string after fence stripping only, without checking that it parses as JSON.
The repo has the correct closing tag. The report's quoted code has a '</<think>' typo that is not in the repo.
json.loads on the unvalidated text can raise JSONDecodeError.
The evaluator logs the error and re-raises it, so malformed JSON fails the evaluation. This is the only caller that propagates the error.
pdf.py already catches the parse error and returns None, so the report's claim of a crash is wrong here.
github.py already catches the parse error and falls back to the first 7 projects, so it does not crash either.
Pull requests
Verdicts are judged from the diff, not run. Merge status is a real git check against e0ece80.
The new ensure_valid_json does brace slicing, a parse check and an LLM repair retry, and evaluator.py and pdf.py both call it, so malformed or prose-wrapped JSON is repaired before json.loads.
extract_json_from_response now validates and returns None, but evaluator.py still raises (ValueError) so the evaluation still fails, the fence-only cleanup is unchanged, and pdf.py loses its brace-slicing rescue because validation now fails first.
It rewrites the extractor to pull the ```json block or the brace-matched object, which handles surrounding prose, but nothing validates the result and evaluator.py still raises on bad JSON.
A regex now strips bare ``` fences, but there is still no validation and prose around the JSON is still not handled.
It adds bare-fence handling and slices to the outermost {…}/[…], which covers surrounding prose, but nothing validates the result and evaluator.py still re-raises on bad JSON.
It slices from the first { to the last }, which fixes stray text around the JSON, but it still does not validate and evaluator.py still re-raises.
It adds one strip() after removing a think block so a fence behind whitespace is recognized, which is a narrow cleanup fix with no validation.
It fixes only the evaluator path, which it retries up to 3 times on JSONDecodeError and trims to the outermost braces; extract_json_from_response is unchanged and the other callers are untouched.
20 PRs that don't fix itShow
It only hardens the GitHub API fetching code in github.py and does nothing about the JSON extraction.
It only takes abs() of deductions.total after json.loads, which has nothing to do with malformed JSON.
It only swaps logger calls for print and tweaks messages; extract_json_from_response and the JSON handling are untouched.
It adds a None check, but extract_json_from_response never returns None in the repo, so the check does nothing and malformed text still reaches json.loads.
It changes PDF text extraction and validation and does not touch JSON parsing.
It reworks PDF section splitting and leaves the JSON extraction and validation alone.
parse_llm_response is a wrapper around the same json.loads and extract_json_from_response, which is unchanged, and it still raises; structured output only lowers the odds for Gemini.
It is mainly about fork detection; the llm_utils.py part adds the same parse_llm_response wrapper, which still raises, and extract_json_from_response is unchanged.
It adds a role-fit scoring call that reuses the same unvalidated extract_json_from_response.
It swaps in the LiteLLM provider and does not change JSON cleanup or validation.
It adds JD scoring and refactors evaluator.py, but extract_json_from_response is unchanged and the JSON parse still raises.
It adds Codeforces scoring and token tracking, and the JSON extraction is untouched.
It adds post-parse bonus and repo-name validation and does not touch JSON extraction.
It only changes a log level and the logging handlers.
It sanitizes PDF text and normalizes scores after parsing, so malformed JSON handling is unchanged.
It adds a web UI and plumbs api_key through, and the JSON extraction is unchanged.
It clamps deductions.total after a successful parse and is unrelated to malformed JSON.
It validates bonus claims and repo names after parsing, and the JSON extraction is untouched.
It adds PR-contributed repo discovery and pinning in github.py and does not touch JSON extraction.
It only changes a log level, the logger source and a message string.
What this packet does not prove
- Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
- PR verdicts are judged from the diff, not run. No PR was applied or tested.
- PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.