Issue #273: Security | Hidden PDF text poisons resume extraction and inflates hiring scores

interviewstreet/hiring-agent · commit e0ece80 · investigation verified · 2026-10-17 · investigation cost $0.09

No open PR fixes this and merges clean

1 fix it but don't merge clean, 1 fix it partly. See the board below.

5/5
citations verified in code
194 → 27
open PRs shortlisted by cited lines
1 fix
0 merge clean · 1 partly
$0.09
investigation cost

How it breaks

The path is the model's reasoning. Each box is a citation checked against the code.

1pdf.py:52 ✓
Extract all PDF text, hidden included
2work.jinja:5 ✓
Raw text injected into section prompts
3pdf.py:143 ✓
Per-section LLMs ingest forged content
4evaluator.py:44 ✓
Evaluator prompt built from poisoned text
5evaluator.py:81 ✓
LLM scores accepted with no grounding
where it shows

What's wrong

The pipeline extracts every text span from the PDF with no visibility check, so hidden white-on-white or tiny text becomes resume content. pdf.py passes that text to the per-section LLM extractors, and the Jinja templates insert it raw, with no trust boundary.
Show full explanationevaluator.py then scores the resulting resume text and builds the evaluation model straight from the LLM's JSON. It does no grounding against the source text or the GitHub API, and applies no score clamp. Forged employers, GSoC and GitHub sections therefore flow into the scores unchanged. Two claims in the report do not match the code. First, evaluator.py does not define or use MAX_BONUS_POINTS or MAX_FINAL_SCORE; a repo-wide grep finds neither. Bonus limits exist only as bonus_max rendered into the prompt. Second, security/poc/ does not exist in this checkout, so the PoC scripts and PDFs could not be inspected or run. I did not read the body of pymupdf_rag.py, so I haven't confirmed that it keeps hidden text; the evidence is that pdf.py calls it on all pages without filtering. I did not run the pipeline, so the score inflation is not reproduced here.
Model confidence: medium.

How to reproduce

  1. Build a 2-page PDF (security/poc/ is absent here, so use PyMuPDF or reportlab). Page 1 holds a visibly weak resume (todo app and calculator). Page 2 holds white, tiny text with forged Google/Meta internships, GSoC, kubernetes contributions and a fake '=== GITHUB DATA ===' block.
  2. Set LLM_PROVIDER=gemini, DEFAULT_MODEL=gemini-2.5-flash and GEMINI_API_KEY, and clear any cached results.
  3. Run the normal pipeline on the PDF: PDFHandler extraction, then convert to text, then ResumeEvaluator.evaluate_resume.
  4. Check that the extracted markdown and JSONResume contain the page-2 hidden content, and that the evaluation cites GSoC/Google/Meta with inflated production/open_source scores.
  5. Control: run a PDF with only the visible page 1 content and compare scores.

Evidence

Each citation was checked against the code at e0ece80.

pdf.py2 citations
line 52
verified in code
resume_text = to_markdown(

Extracts all pages with no filtering of invisible (white-on-white or tiny) text, so hidden spans enter the pipeline.

line 143
verified in code
prompt = self.template_manager.render_template("work", text_content=resume_text)

The unsanitized extracted text is passed straight to the per-section LLM extractor.

prompts/templates/work.jinja1 citation
line 5
verified in code
{{ text_content }}

The full extracted text is inserted raw into the prompt, with no untrusted-data boundary or grounding rule. The other section templates do the same.

evaluator.py2 citations
line 44
verified in code
self.role.criteria_source, text_content=resume_text

The evaluation prompt is built from the poisoned resume text, with no check against the visible layer or the GitHub API.

line 81
verified in code
evaluation_data = self.evaluation_model(**evaluation_dict)

The LLM's JSON is accepted as-is, with no grounding or clamping of scores in code. The MAX_BONUS_POINTS and MAX_FINAL_SCORE constants named in the report are not in the repo.

Pull requests

Verdicts are judged from the diff, not run. Merge status is a real git check against e0ece80.

194open PRs
27change a cited line or mention the issue
2change the cause
0fix and merge clean
#275 and #450 all change the cause. Once one is merged, the others duplicate it.
Merges clean
Conflicts or unknown
Fixes
—
Partly
—
Why each verdict

It swaps to_markdown for visible-span-only extraction in pdf.py, strips forged GITHUB DATA blocks, and in evaluator.py normalizes the LLM output with source-text/GitHub grounding and score clamping, which covers the extraction, trust-boundary and grounding/clamp gaps.

Shortlisted because: mentions the issue, changes pdf.py:52

It drops spans whose color matches the page background inside pymupdf_rag, which removes white-on-white text, but tiny/off-page/other hidden text, the raw template insertion, and the missing grounding/clamping in evaluator.py remain.

Shortlisted because: mentions the issue
25 PRs that don't fix itShow

It only edits the work prompt rules and appends the text again as a user message, which adds no trust boundary or filtering.

Shortlisted because: changes prompts/templates/work.jinja:5

It only adds an optional job_description parameter to evaluation, with no change to the poisoned-text handling.

Shortlisted because: changes evaluator.py:44
#41 · Fix issue #40 by Anubhav200311

It only splits the text into sections for token savings; extraction still takes all spans, the hidden text is still passed to the LLMs, and evaluator.py is untouched.

Shortlisted because: changes pdf.py:143, changes prompts/templates/work.jinja:5

It extracts per-section snippets by regex from the same unfiltered text, so hidden spans still pass through and evaluation is unchanged.

Shortlisted because: changes pdf.py:143

It only replaces logger calls with print statements and does not touch extraction, templates or evaluation.

Shortlisted because: changes evaluator.py:81, changes pdf.py:143

It adds certificate and volunteer extractors that use the same unfiltered text and raw template insertion.

Shortlisted because: changes pdf.py:52

It only adds None handling for malformed JSON and does nothing about hidden text or output validation.

Shortlisted because: changes evaluator.py:81

It validates the file extension, MIME type and size, but the content of the PDF is still extracted unfiltered.

Shortlisted because: changes pdf.py:52

It moves settings into a config file, and its MAX_* constants in evaluator.py are only defined there and never applied to clamp the output.

Shortlisted because: changes pdf.py:52

It closes the document and adds validation, with no change to what text is extracted or how scores are checked.

Shortlisted because: changes pdf.py:52

It only adds a None check on the extracted JSON and a log-level change.

Shortlisted because: changes evaluator.py:81

It switches to page.get_text('text'), which still extracts every span including invisible ones, and adds no visibility filtering, grounding or clamping.

Shortlisted because: changes pdf.py:143, changes pdf.py:52

It reworks header detection and section splitting but still extracts all text with no visibility check, and evaluation is unchanged.

Shortlisted because: changes pdf.py:143, changes pdf.py:52

It switches to structured output parsing, which enforces the schema but not the truth of the content, and has no visibility filtering, grounding or clamping.

Shortlisted because: changes evaluator.py:81

Its evaluator.py and pdf.py changes only switch to structured output parsing, and the contributors-API change in github.py concerns fork detection, not grounding of the resume claims.

Shortlisted because: changes evaluator.py:81

It adds job description analysis and a job_match_score and does not touch hidden text or grounding.

Shortlisted because: changes evaluator.py:44

It adds JD-augmented scoring and batch processing, and still builds the evaluation from raw LLM JSON with no grounding or clamping.

Shortlisted because: changes evaluator.py:44, changes evaluator.py:81

It adds retries, JSON repair, caching and parallelism, and the hidden text still flows through with the LLM JSON accepted unchecked.

Shortlisted because: changes evaluator.py:81

It only makes the role and organization configurable in the evaluation template.

Shortlisted because: changes evaluator.py:44

It only changes a log level and logging stream routing.

Shortlisted because: changes evaluator.py:81

It only adds PoC PDFs, scripts and write-ups under security/ and explicitly states it implements no fix.

Shortlisted because: mentions the issue

It passes role-based score weights into the prompt, which are only prompt text and not enforced by clamping or grounding in code.

Shortlisted because: changes evaluator.py:44

It adds a provider and retry logic for invalid JSON in evaluator.py, with no sanitization, grounding or score clamp.

Shortlisted because: changes evaluator.py:44, changes evaluator.py:81

It expands the extraction prompts, but the text is still inserted raw with no untrusted-data boundary, and the extraction and evaluation code are unchanged.

Shortlisted because: changes prompts/templates/work.jinja:5

It adds ignore_graphics=True to to_markdown so more text is kept, which makes hidden-text extraction no better and arguably worse.

Shortlisted because: changes pdf.py:52

What this packet does not prove

  • Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
  • PR verdicts are judged from the diff, not run. No PR was applied or tested.
  • PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.