Issue #271: Backend projects are penalized for missing live demos

interviewstreet/hiring-agent · commit e0ece80 · investigation verified · 2026-10-16 · investigation cost $0.05

No open PR fixes this and merges clean

4/4
citations verified in code
194 → 2
open PRs shortlisted by cited lines
0 fix
0 merge clean · 0 partly
$0.05
investigation cost

How it breaks

The path is the model's reasoning. Each box is a citation checked against the code.

1criteria.jinja:82 ✓
Rubric treats live demo as universal expectation
2criteria.jinja:84 ✓
Live demo bonus favors UI projects
3criteria.jinja:172 ✓
GitHub-only projects lose 2-3 points
4criteria.jinja:171 ✓
Backend projects penalized despite strong repos
where it shows

What's wrong

The scoring rubric in the software_engineering_intern role templates penalizes every project that has no live demo, whatever its type. It has no exception for backend, infrastructure, CLI or library projects, whose code is the deliverable. A GitHub-only project loses 2-3 points in the deduction list.
Show full explanationThe link rules also cut the score of projects with no links and add a bonus for projects with a live demo. I found no code that checks project type before these rules apply. They are plain prompt text that the LLM follows literally. Line 26 of system_message.jinja also mentions live demos, but I could not quote it because the line is too long.
Model confidence: medium.

How to reproduce

  1. Prepare a resume whose projects are backend-only, such as a REST API, Kafka pipeline or CLI tool. Give each a GitHub link and no hosted demo.
  2. Run the evaluation pipeline with the software_engineering_intern role, which renders roles/software_engineering_intern/criteria.jinja and system_message.jinja into the prompt.
  3. Check the project score and the deductions in the output. Each GitHub-only project loses about 2-3 points for having no live demo.
  4. Add a hosted demo URL to the same resume and rerun. The score goes up, and the 10-20% live demo bonus applies.

Evidence

Each citation was checked against the code at e0ece80.

roles/software_engineering_intern/criteria.jinja4 citations
line 172
verified in code
- -2 to -3 points for each project with only GitHub link but no live demo

A numeric deduction for having no live demo, applied to every project with no check on project type.

line 171
verified in code
- -3 to -5 points for each project without any GitHub link, live demo, or active URL

The link deductions treat a live demo as an expected deliverable for all projects, not just UI ones.

line 84
verified in code
- **LIVE DEMO BONUS**: Projects with working live demos should receive 10-20% higher scores

A bonus for live demos that backend projects cannot realistically earn, so they score lower relative to UI projects.

line 82
verified in code
- **NO LINKS**: Projects without URLs, GitHub links, or live demos should receive 30-50% lower scores

Lists live demos as one of the links a project needs, with no carve-out for backend or infrastructure work.

Pull requests

Verdicts are judged from the diff, not run. Merge status is a real git check against e0ece80.

194open PRs
2change a cited line or mention the issue
0change the cause
0fix and merge clean
2 PRs that don't fix itShow

It edits prompts/templates/resume_evaluation_criteria.jinja (plus .gitignore) and never touches roles/software_engineering_intern/criteria.jinja, where the live-demo deductions and bonus at lines 82-84 and 171-172 live, so that rubric is unchanged.

Shortlisted because: mentions the issue

It only edits prompts/templates/resume_evaluation_criteria.jinja, a different file from the cited roles/software_engineering_intern/criteria.jinja, so the intern role's demo deduction and link rules remain.

Shortlisted because: mentions the issue

What this packet does not prove

  • Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
  • PR verdicts are judged from the diff, not run. No PR was applied or tested.
  • PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.