Issue #273: Security | Hidden PDF text poisons resume extraction and inflates hiring scores
No open PR fixes this and merges clean
1 fix it but don't merge clean, 1 fix it partly. See the board below.
How it breaks
The path is the model's reasoning. Each box is a citation checked against the code.
What's wrong
The pipeline extracts every text span from the PDF with no visibility check, so hidden white-on-white or tiny text becomes resume content. pdf.py passes that text to the per-section LLM extractors, and the Jinja templates insert it raw, with no trust boundary.Show full explanation
evaluator.py then scores the resulting resume text and builds the evaluation model straight from the LLM's JSON. It does no grounding against the source text or the GitHub API, and applies no score clamp. Forged employers, GSoC and GitHub sections therefore flow into the scores unchanged. Two claims in the report do not match the code. First, evaluator.py does not define or use MAX_BONUS_POINTS or MAX_FINAL_SCORE; a repo-wide grep finds neither. Bonus limits exist only as bonus_max rendered into the prompt. Second, security/poc/ does not exist in this checkout, so the PoC scripts and PDFs could not be inspected or run. I did not read the body of pymupdf_rag.py, so I haven't confirmed that it keeps hidden text; the evidence is that pdf.py calls it on all pages without filtering. I did not run the pipeline, so the score inflation is not reproduced here.How to reproduce
- Build a 2-page PDF (security/poc/ is absent here, so use PyMuPDF or reportlab). Page 1 holds a visibly weak resume (todo app and calculator). Page 2 holds white, tiny text with forged Google/Meta internships, GSoC, kubernetes contributions and a fake '=== GITHUB DATA ===' block.
- Set LLM_PROVIDER=gemini, DEFAULT_MODEL=gemini-2.5-flash and GEMINI_API_KEY, and clear any cached results.
- Run the normal pipeline on the PDF: PDFHandler extraction, then convert to text, then ResumeEvaluator.evaluate_resume.
- Check that the extracted markdown and JSONResume contain the page-2 hidden content, and that the evaluation cites GSoC/Google/Meta with inflated production/open_source scores.
- Control: run a PDF with only the visible page 1 content and compare scores.
Evidence
Each citation was checked against the code at e0ece80.
Extracts all pages with no filtering of invisible (white-on-white or tiny) text, so hidden spans enter the pipeline.
The unsanitized extracted text is passed straight to the per-section LLM extractor.
The full extracted text is inserted raw into the prompt, with no untrusted-data boundary or grounding rule. The other section templates do the same.
The evaluation prompt is built from the poisoned resume text, with no check against the visible layer or the GitHub API.
The LLM's JSON is accepted as-is, with no grounding or clamping of scores in code. The MAX_BONUS_POINTS and MAX_FINAL_SCORE constants named in the report are not in the repo.
Pull requests
Verdicts are judged from the diff, not run. Merge status is a real git check against e0ece80.
It swaps to_markdown for visible-span-only extraction in pdf.py, strips forged GITHUB DATA blocks, and in evaluator.py normalizes the LLM output with source-text/GitHub grounding and score clamping, which covers the extraction, trust-boundary and grounding/clamp gaps.
It drops spans whose color matches the page background inside pymupdf_rag, which removes white-on-white text, but tiny/off-page/other hidden text, the raw template insertion, and the missing grounding/clamping in evaluator.py remain.
25 PRs that don't fix itShow
It only edits the work prompt rules and appends the text again as a user message, which adds no trust boundary or filtering.
It only adds an optional job_description parameter to evaluation, with no change to the poisoned-text handling.
It only splits the text into sections for token savings; extraction still takes all spans, the hidden text is still passed to the LLMs, and evaluator.py is untouched.
It extracts per-section snippets by regex from the same unfiltered text, so hidden spans still pass through and evaluation is unchanged.
It only replaces logger calls with print statements and does not touch extraction, templates or evaluation.
It adds certificate and volunteer extractors that use the same unfiltered text and raw template insertion.
It only adds None handling for malformed JSON and does nothing about hidden text or output validation.
It validates the file extension, MIME type and size, but the content of the PDF is still extracted unfiltered.
It moves settings into a config file, and its MAX_* constants in evaluator.py are only defined there and never applied to clamp the output.
It closes the document and adds validation, with no change to what text is extracted or how scores are checked.
It only adds a None check on the extracted JSON and a log-level change.
It switches to page.get_text('text'), which still extracts every span including invisible ones, and adds no visibility filtering, grounding or clamping.
It reworks header detection and section splitting but still extracts all text with no visibility check, and evaluation is unchanged.
It switches to structured output parsing, which enforces the schema but not the truth of the content, and has no visibility filtering, grounding or clamping.
Its evaluator.py and pdf.py changes only switch to structured output parsing, and the contributors-API change in github.py concerns fork detection, not grounding of the resume claims.
It adds job description analysis and a job_match_score and does not touch hidden text or grounding.
It adds JD-augmented scoring and batch processing, and still builds the evaluation from raw LLM JSON with no grounding or clamping.
It adds retries, JSON repair, caching and parallelism, and the hidden text still flows through with the LLM JSON accepted unchecked.
It only makes the role and organization configurable in the evaluation template.
It only changes a log level and logging stream routing.
It only adds PoC PDFs, scripts and write-ups under security/ and explicitly states it implements no fix.
It passes role-based score weights into the prompt, which are only prompt text and not enforced by clamping or grounding in code.
It adds a provider and retry logic for invalid JSON in evaluator.py, with no sanitization, grounding or score clamp.
It expands the extraction prompts, but the text is still inserted raw with no untrusted-data boundary, and the extraction and evaluation code are unchanged.
It adds ignore_graphics=True to to_markdown so more text is kept, which makes hidden-text extraction no better and arguably worse.
What this packet does not prove
- Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
- PR verdicts are judged from the diff, not run. No PR was applied or tested.
- PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.