HackerRank open-sourced its resume-scoring agent last week and engineers started running their own CVs through it. The results were immediately strange. One engineer scored 90/100 on the first run, cleaned up some debug print statements, and got 74/100. Same resume. Same system. The only change was deleting lines that existed purely for local development. He ran it a hundred more times: scores ranged from 66 to 99. If a company’s cutoff is 85, he fails 65% of the time through pure chance.
That story — originally from Dan Kinsky on Substack, now sitting at nearly 700 points on Hacker News — is worth unpacking in detail, because the failure mode here is not a bug. It’s a fundamental architectural choice made by engineers who appear to understand LLM non-determinism in the abstract, but then built a scoring pipeline that depends on deterministic LLM behavior in practice. The repo is called hiring-agent. It has 3,578 stars and 742 forks. Hundreds of companies are likely running something structurally similar right now, even if they didn’t fork this specific repository.
This is not an attack on HackerRank. What they published is more honest than what most HR tech vendors ship — it’s open source, it’s readable, and the architecture is straightforward enough that you can understand exactly where the variance comes from. That transparency is what makes it worth studying. The problem isn’t that HackerRank built something bad. The problem is that the entire category of “LLM-as-judge for hiring” has a variance problem that nobody has solved, and most vendors are papering over it with false precision.
What the Hiring Agent Actually Does
Before getting into the failure modes, it’s worth understanding the architecture, because it’s actually well-designed for what it’s doing in the first three-quarters of the pipeline.
The pipeline has five stages:
- PDF extraction: PyMuPDF converts the resume to Markdown-like text, handling headings, links, tables, and formatting. This is deterministic — same PDF, same output every time.
- Section parsing: The LLM is called once per section (basics, work history, education, skills, projects, awards) using Jinja templates. It extracts structured JSON. This is mostly deterministic for factual fields — name, company, dates, skills listed explicitly.
- GitHub enrichment: The system pulls your GitHub profile if one is linked, fetches repos, and asks the LLM to select the top 7 projects by contribution signal. This introduces some variance but is capped by the repo list itself.
- Evaluation: A single LLM call with the assembled structured data produces category scores. This is where variance explodes.
- Output: Scores are printed and optionally written to CSV.
The scoring breakdown is:
- 35 points: Open source contributions
- 30 points: Personal projects
- 25 points: Work experience
- 10 points: Technical skills
- Up to 20 bonus points: Startup experience, portfolio site, technical blog, etc.
The maximum is 120 points (100 base + 20 bonus), which gets normalized or displayed as a score out of 100 plus potential bonus. The default LLM is gemma3:4b running locally via Ollama at temperature 0.1.
What does temperature 0.1 mean in practice? Temperature controls the randomness of the next-token selection. At 0, the model greedily picks the highest-probability token at every step — theoretically deterministic, though floating-point arithmetic and CUDA kernel ordering mean even temperature=0 can produce slightly different outputs across runs on different hardware. At 0.1, there’s a small amount of intentional randomness added back. The README describes this as “low, supposedly nudging the model toward deterministic outputs.” The word “supposedly” is doing a lot of work there.
Where the Variance Actually Lives
Dan Kinsky’s analysis breaks the variance down by category, and the pattern is instructive:
Technical skills: Near-perfect consistency. 8/10 in 98 out of 100 runs. This makes sense — technical skills are a checklist. You either have “Python” in your resume or you don’t. The LLM is pattern-matching explicit keywords against an explicit list. The expected output space is tiny, and the correct answer is mostly deterministic.
Open source: Became consistent after recent commits to the repo. The category was improved to have clearer rubrics, and the GitHub data provides concrete evidence — stars, commits, repo language — that anchors the LLM’s judgment.
Work experience: Perfect 25/25 in every single run. A junior engineer with one internship gets 25/25. A principal with a decade of distributed systems gets 25/25. This sounds good (consistent!) but it’s a different kind of failure — the rubric is so underspecified that the LLM just gives full marks to everyone. Two lines in the evaluation template, no anchors, no examples. Consistent, but useless.
Projects: Wild variance. This is the 30-point category, and it’s where scores swing dramatically between runs. One run says the projects “lack architectural complexity.” The next says they “demonstrate real-world deployment.” Same projects. Same description. The LLM is making a judgment call that the prompt doesn’t constrain tightly enough.
The reason projects show the most variance is that the prompt asks for something that doesn’t have a ground truth answer. “Rate this project 0-30 based on its engineering quality” is a fundamentally open-ended question. The LLM has seen millions of takes on what makes software engineering “quality” and has no way to anchor which perspective applies here. Every run samples a slightly different path through its probability distribution.
Temperature=0 Doesn’t Fix This
Someone opened a GitHub issue in October 2025 showing scores of 27, 34, 32, 34, 34, 30 across six consecutive runs at temperature=0. This is the key insight that breaks the naive intuition that “just lower the temperature” is a fix.
Why doesn’t temperature=0 produce identical results?
First, hardware non-determinism. GPU floating-point operations are not deterministic by default. CUDA kernels run in different orders across threads. For transformer inference, this means that even with identical inputs and temperature=0, you can get slightly different logit values due to floating-point rounding in the attention computation, which changes which token gets the highest probability at key decision points.
Second, context length sensitivity. The scoring prompt includes the full assembled resume, GitHub data, and evaluation criteria. Small differences in how the GitHub enrichment step renders the top 7 repos (different line ordering, different formatting of commit counts) change the exact byte sequence fed to the LLM, which changes the activation patterns, which changes the output.
Third, and most fundamentally: even if you somehow achieved perfect determinism at temperature=0, you’d still have the underlying problem that the prompt doesn’t constrain the output space enough. “Consistent” and “correct” are different things. If the model consistently gives bad rubric application, you’ve just made the bad judgment repeatable.
Let me reproduce the core issue with a minimal example. Here’s a simplified version of what the evaluation template is doing:
### Projects (0-30 points)
Evaluate the candidate's personal and open-source projects.
Consider:
- Technical sophistication
- Real-world deployment evidence
- Architectural complexity
- Innovation and novelty
Projects:
{{ candidate_projects }}
Score this section with a number between 0 and 30 and provide a brief justification.This template has four evaluation axes and one numeric output range. It doesn’t define what “technical sophistication” means at the 10-point level versus the 20-point level. It doesn’t give examples. It doesn’t anchor the distribution. A rubric like this produces high variance because the model’s priors about “what deserves 15 points vs 22 points” are drawn from its training distribution, which contains wildly different standards across different engineering cultures, company sizes, and time periods.
Compare this to the technical skills template, which presumably looks more like: “Check whether the candidate has each of the following skills: [list]. Award 1 point per match, maximum 10.” That’s a different kind of task — it’s information retrieval, not judgment. The output space is bounded and deterministic.
The Structural Problem: 65% of the Score Requires Judgment
Here’s the math that matters for hiring decisions: 35 points (open source) + 30 points (projects) = 65% of the base score requires LLM judgment calls. The remaining 35% (work experience + technical skills) is either a checklist or apparently evaluated with a rubric so loose it gives everyone full marks.
This means that if your company uses this tool with a cutoff of 85, you’re not screening for engineering quality. You’re screening for engineering quality plus a random draw from the LLM’s opinion distribution on the day your resume gets processed.
The 65% weighting on open source and projects specifically disadvantages candidates who built things that never ended up on GitHub. This is a well-documented demographic pattern in software engineering: engineers at large companies frequently sign NDAs or work on internal systems where the code can’t be shared. Senior engineers working on classified government systems, financial infrastructure, or enterprise software at companies with strong IP protection may have decades of high-quality engineering work that’s entirely invisible to this scoring system.
Someone who built the payments infrastructure at a major bank for fifteen years and has a GitHub profile with three weekend projects will score lower than a recent graduate with ten starred repos. The work experience category is worth 25 points with a rubric that apparently gives full marks regardless of depth. The GitHub profile is worth 35 points and directly visible to the model.
I want to be careful here: this criticism applies to the default configuration and the documented scoring weights. The tool is open source, the weights are configurable, and nothing stops a company from changing them. But the defaults signal a philosophical assumption: that publicly visible code is the primary evidence of engineering quality. That assumption was always questionable and becomes actively harmful when embedded in automated screening infrastructure.
What “Explainable” Really Means Here
The README describes the tool as producing “a fair, explainable evaluation.” Explainability is a term of art in AI systems that means different things in different contexts. Here it means the model produces a text justification alongside the numeric score. One run says “Projects demonstrate strong architectural thinking with evidence of production deployment.” Another run says “Projects lack the architectural sophistication expected at this level.”
These justifications are post-hoc rationalizations of whatever score the model sampled. They sound plausible because the LLM is good at generating plausible-sounding text. But they don’t reflect a stable underlying judgment process — they reflect whichever path through the probability tree the model happened to take on that particular inference.
This is a known failure mode of LLM self-explanation. Models don’t have access to their own computation graph. When asked to explain a decision, they generate text that sounds like an explanation using the same next-token prediction process they use to generate everything else. The explanation is conditioned on the score, not derived from the same reasoning that produced it.
In practice this means you can get a high score with a confident-sounding explanation that says “demonstrates deep architectural expertise,” and a low score with an equally confident-sounding explanation that says “lacks evidence of architectural depth.” Both explanations will seem reasonable to a hiring manager who doesn’t know the resume scored 99 one hour ago and 66 the next.
How This Compares to the Broader ATS Market
HackerRank’s hiring-agent is unusually honest about what it is because it’s open source and the failure modes are visible. The closed-source ATS systems from Greenhouse, Lever, Workday, and the dozens of AI-enhanced screening vendors that have launched in the past two years are doing the same thing behind an API and a vendor dashboard with no public variance testing.
I’ve seen vendor demos for AI resume screening tools where the sales pitch includes phrases like “consistent, bias-free evaluation” and “objective scoring.” The demos show a single resume getting scored. Nobody runs the same resume a hundred times in the demo. Nobody checks the variance. The tool produces a number with two decimal places, which signals precision, and precision signals objectivity.
The decimal places are particularly insidious. If a system tells you a candidate scored 83, you might treat that as approximately equivalent to 80 or 85. If it tells you they scored 83.47, you start treating 83.47 as meaningfully different from 83.12. The precision of the output is decoupled from the precision of the underlying judgment. LLM-based scoring systems with temperature > 0 are literally incapable of producing a true 83.47 score — they produce scores that, if you ran them enough times, would be drawn from some distribution centered somewhere in the 70s-90s. The 83.47 is one sample from that distribution, not a measurement of a real quantity.
There’s a useful analogy here to measuring instruments. A thermometer that displays six decimal places is not necessarily more accurate than one that displays two — if the sensor has noise, adding decimal places to the display just adds false precision. ATS systems with LLM scoring components are adding decimal places to a noisy sensor. The noise doesn’t go away because you display more digits.
The Temperature 0.1 Choice Is Not Arbitrary
I want to spend a moment on the choice of temperature=0.1 specifically, because it suggests the developers understood that lower temperatures would reduce variance but chose not to go to zero.
There are legitimate reasons not to use temperature=0 even when you want near-deterministic outputs:
Diversity of expression: At temperature=0, the model always produces the same phrasing. If you’re generating justifications that get logged and reviewed by humans, identical justifications for different candidates would look suspicious and mechanical. A small temperature adds just enough variation to make the explanations look “naturally” different while keeping them roughly in the same ballpark.
Avoiding mode collapse: Some models, particularly smaller quantized models like gemma3:4b, can get stuck in repetitive loops at very low temperatures. A tiny amount of randomness prevents this.
Practical experience: Temperature=0 sometimes produces worse outputs for certain prompt types than temperature=0.1. The slight randomness can help the model avoid “local optima” in its generation that produce verbose but empty responses.
None of these reasons are wrong, but they all implicitly accept that the scoring output will have variance. The design accepts non-determinism as a trade-off for other properties. The problem is that this trade-off is invisible to end users who see a score and assume it represents a stable measurement.
Can You Fix This?
Partially. Here’s what actually helps and what doesn’t:
What doesn’t help: lowering temperature. The community has already proven this. Temperature=0 still produces scores of 27, 30, 32, 34, 34, 34 across six runs. The variance comes from the rubric underspecification and model stochasticity at the floating-point level, not primarily from temperature sampling.
What helps some: better rubrics. The technical skills category scores consistently because the rubric is a checklist. If you redesigned the projects and open source categories to have the same kind of explicit, bounded criteria — “award 5 points if the repo has >50 stars, 10 points if >500, 15 points if >5000” — you’d get more consistent scores. You’d also be measuring something less nuanced, but that’s the trade-off.
What helps more: majority vote / ensemble scoring. Run the evaluation N times and take the median or mode. With N=10 runs, the variance in the sample mean is reduced by sqrt(10) relative to a single run. This is more expensive but directly addresses the sampling variance. The open-source contributions category appears to have improved through this approach based on community reports.
What helps most: separate extraction from judgment. The pipeline should explicitly separate “what is factually in this resume” from “how should we score these facts.” Factual extraction (name, companies, years, technologies) should use very low or zero temperature. Scoring should be redesigned so that the judgment component is minimal or eliminates — ideally just matching facts against explicit criteria tables, not asking the LLM to evaluate subjective quality.
BETTER_APPROACH = {
"extraction": {
"temperature": 0.0,
"task": "Parse these fields exactly as stated in the resume",
"output": "structured JSON with no judgment",
},
"scoring": {
"temperature": 0.0,
"task": "Apply these explicit rules to the extracted data",
"rules": [
"open_source_stars_0_to_50: 5 pts",
"open_source_stars_50_to_500: 10 pts",
"open_source_stars_over_500: 20 pts",
"work_experience_years_0_to_2: 5 pts",
"work_experience_years_2_to_5: 15 pts",
"work_experience_years_over_5: 25 pts",
],
"output": "apply rules mechanically, no discretion"
}
}
This approach would be more consistent. It would also be less interesting, and the rules would encode explicit value judgments that are currently hidden inside the LLM’s priors. How many GitHub stars qualifies as “significant open source contribution”? Whoever writes the scoring table is making that call, visibly, in a way that can be audited and challenged. That’s actually better than having the LLM make the same call invisibly and inconsistently.
What helps systemically: using different models for different tasks. gemma3:4b is a 4-billion-parameter model designed for local inference. It’s impressive for its size, but smaller models generally have higher variance on open-ended judgment tasks than larger models. Switching to a larger model (the repo supports Gemini, and the author tested gemini-2.0-flash) reduced the score range from 66-99 to 48-64. That’s better variance, but the author correctly notes: if your cutoff is 60, you’re still failing 28% of the time through no fault of your own.
The Deeper Problem: What We’re Actually Optimizing For
There’s a meta-question worth asking: even if you fixed the variance, should automated resume scoring be doing what this tool is trying to do?
The tool’s 65% weighting on publicly visible code reflects a specific theory of engineering quality: that the best engineers have active GitHub profiles and significant open source contributions. This theory is not obviously wrong for certain types of roles — developer tools companies, open source foundations, startups that build on open source infrastructure. It’s significantly wrong for other types: enterprise software, regulated industries, defense contractors, large financial institutions.
The tool also gives 25 points for work experience with a rubric that, based on the testing results, doesn’t actually differentiate. The “fairness constraints” mentioned in the README’s description of `evaluator.py` are doing unknown work. If the fairness constraints cause the model to give full work experience marks to everyone regardless of actual experience, that’s not fairness — it’s the rubric being so vague that the model defaults to maximum scores.
The GitHub enrichment step specifically favors candidates who have public GitHub profiles. A candidate without a GitHub link in their resume gets no enrichment data, and depending on how the model weights absence of evidence, this could be interpreted as evidence of lower open source contribution. The tool doesn’t appear to give explicit credit for “no GitHub found” — it simply has no GitHub data to enrich the evaluation with. If your company primarily hires engineers who don’t have public profiles, this category systematically undersells your candidate pool.
None of this is unique to HackerRank. These are industry-wide assumptions baked into most technical resume screening systems: value public code, value visible side projects, value demonstrated production deployment experience. The assumptions became industry-standard when Google and Facebook started screening for them in the early 2010s, and the ATS software industry has largely codified them. What’s new is that LLMs apply these assumptions with variable precision and no ability to explain their variance.
What Hiring Managers Should Actually Do
If you’re an engineer who has any influence over how your company handles resume screening, here’s the practical guidance:
Test your screening tool for variance. Run the same resume through ten times and check whether the scores are stable. If they vary by more than 5-10 points, you have a variance problem that may be producing different results for different batches of candidates depending on nondeterministic factors.
Don’t use a fixed numeric cutoff on LLM-scored applications. If a tool scores candidates 0-100 and you reject anyone below 80, you’re making binary decisions based on noisy measurements. At minimum, use a band — treat scores of 75-85 as roughly equivalent, require human review for borderline cases, and reserve automated rejection only for clear outliers at the bottom of the distribution.
Audit the rubric explicitly. For any LLM-based scoring system (in-house or vendor), ask for the actual evaluation prompt. If the vendor won’t show you the prompt, that’s information. If they do show you the prompt and it has ambiguous open-ended questions, treat the output as a rough signal, not a measurement.
Separate factual extraction from quality judgment. LLMs are excellent at parsing unstructured resumes into structured fields. They’re much worse at consistently judging the quality of what they parsed. Use LLMs for the extraction step, and keep humans or rule-based systems in the loop for the quality judgment step.
Be explicit about what your rubric is optimizing for. If your scoring system weights GitHub stars heavily, acknowledge that you’re optimizing for candidates who prioritize public open source work. That’s a legitimate filter for some roles. It’s a potentially bad filter if you’re hiring for enterprise software, regulated industries, or senior engineers who spent their careers building non-public infrastructure.
Run regular calibration checks. Periodically take a set of resumes from candidates you’ve already hired and found to be strong performers, and run them through your screening tool. If the tool consistently gives your best engineers borderline scores, your calibration is off.
The Broader Stakes
HackerRank’s hiring-agent is a well-built, honest implementation of a category of software that’s about to be deployed at massive scale. The fact that it’s open source means engineers can actually see the variance. The closed-source commercial equivalents have the same variance and no transparency.
In the United States, employment discrimination law requires that screening criteria be job-related and consistent in application. A screening tool that produces significantly different scores for the same candidate across runs has a consistency problem that could expose companies to legal risk — not because the tool is biased in the protected-characteristic sense, but because it’s producing random results dressed up as objective measurement.
The EU AI Act classifies AI systems used in employment decisions as high-risk, requiring human oversight, audit trails, and documentation of performance. A tool with the variance profile documented here — 66 to 99 across 100 runs on the same input — would need significant guardrails to meet that standard. Most companies deploying commercial ATS systems with LLM components are not documenting the variance of those systems, and may not know what it is.
The engineers who built hiring-agent didn’t build something irresponsible. They built something transparent, and transparency revealed a real problem. The irresponsible versions are the ones sitting inside vendor contracts that nobody tested for variance, scoring candidates at scale, with hiring managers reading the output as precise measurements.
The score your resume gets from an AI screener is not a measurement of your ability. It’s a sample from a distribution the company probably hasn’t characterized. Whether you clear the cutoff or not may depend more on the random state of the inference engine on the day your application was processed than on anything you actually built or accomplished. The tool that makes this visible deserves more engagement than the tools that hide it behind dashboards with decimal places.
Dan Kinsky ended his post with a sharp observation: “You might as well throw out half the resumes and tell the applicants you don’t fuck with bad luck.” That’s hyperbole, but it’s the right direction. The variance in LLM resume scoring is not noise around a true signal — it is the signal that the measurement is insufficiently specified to produce reliable results. Fix the rubrics, test for variance, use humans for judgment calls, and stop treating LLM scores as the thing they look like but are not.
A Reproducible Test for Any ATS System
If you want to audit any LLM-based screening tool — whether it’s hiring-agent, a commercial vendor, or an in-house build — here’s a minimal protocol that takes about an hour and reveals the variance profile:
import subprocess
import statistics
def test_ats_variance(resume_path, n_runs=20):
scores = []
for i in range(n_runs):
result = subprocess.run(
["python", "score.py", resume_path],
capture_output=True, text=True
)
score = parse_score_from_output(result.stdout)
scores.append(score)
print(f"Run {i+1}: {score}")
print(f"\nMean: {statistics.mean(scores):.1f}")
print(f"Median: {statistics.median(scores):.1f}")
print(f"Std dev: {statistics.stdev(scores):.1f}")
print(f"Min: {min(scores)}, Max: {max(scores)}")
print(f"Range: {max(scores) - min(scores)}")
print(f"95th percentile: {sorted(scores)[int(0.95 * n_runs)]}")
print(f"5th percentile: {sorted(scores)[int(0.05 * n_runs)]}")
return scores
Run this with your own resume (or a synthetic one). If the range is larger than 15 points, the tool is not making a measurement — it’s rolling dice. If the standard deviation is larger than 5 points, any individual score is only meaningful to within ±10 points at 95% confidence. A cutoff that’s within one standard deviation of your score means you’re at genuine risk of being filtered out by chance.
Run it with multiple resumes: one from your strongest hire in the last year, one from a rejected candidate who turned out to be talented, one from a mediocre performer you regret hiring. If the tool’s scores track your hindsight judgment consistently across all three, it might be worth using. If the scores are scattered across the range for all three, the tool is not predictive — it’s producing random numbers in a plausible range.
This kind of variance testing is standard practice in measurement science and almost completely absent from HR tech evaluation. Companies spend months negotiating contract terms with ATS vendors without ever asking for a variance report. They accept demos that show single-point precision. They deploy at scale. The tools fail in the same way hiring-agent fails, just invisibly.
The Irony of Publishing This
There’s an important point worth making about what HackerRank did by open-sourcing this. Publishing the code created accountability that closed-source systems don’t have. Dan Kinsky could run the variance test because the code was public and he could install it locally. He could read the evaluation template and see that the work experience rubric was two lines with no anchors. He could file a GitHub issue and have it acknowledged and iterated on.
That’s how software should work. The problem is that the publication also created a reputational surface — “HackerRank’s ATS is broken” — while the real story is “all LLM-based ATS systems have this problem.” HackerRank is being held accountable for being transparent. The vendors who haven’t published anything are not being held accountable at all.
This dynamic discourages future transparency. If open-sourcing a tool creates negative press while keeping it closed avoids scrutiny, the rational choice for future companies is to keep tools closed. That’s bad for the industry, bad for candidates, and bad for the engineers who want to understand what’s actually being measured when they apply for jobs.
The right response to the HackerRank hiring-agent situation is not to criticize them for building a flawed tool. It’s to use their transparency as a forcing function: demand the same variance documentation from every vendor, make it standard practice to publish variance tests alongside benchmark numbers, and treat “consistent and objective” marketing claims as requiring proof rather than assumption.
Every company that deploys an AI resume screener should be able to answer: what is the standard deviation of this tool’s scores across 50 runs on the same resume? If they can’t answer that question, they’re deploying a black box with false precision. The box isn’t objective. It’s just opaque.
Discussion