Evaluating the AI Shortlist
Your AI agent ranked three hundred CVs. Now see why, finalist by finalist.
An AI screening agent hands back a ranked shortlist and a score for each criterion. The decision still needs a person who can see what every score rests on. This knowledge graph puts six finalists for a data engineering role, their scorecard and the evidence in one place. Click a criterion and every finalist's AI score, human score and the reason behind each sit side by side.
A ranking tells you who. Your decision needs why.
AI screening saves the first pass through hundreds of CVs, but few recruiters take its score at face value. In a small study of 25 recruiters by Enhancv, 44% had scoring tools and more than half of those switched them off or ignored them; only 8% treated the score as decisive (BetaNews, November 2025).
The law asks for the same care. Under the GDPR, the EU Court of Justice has held that a score which drives the outcome can count as the decision itself, even when a person signs it off (the SCHUFA ruling). In New York City, an automated hiring tool needs a yearly bias audit and notice to candidates (Local Law 144). Either way, the person deciding has to be able to read the evidence, not only the rank.
Every score next to its evidence
The hiring manager's eight criteria sit around the role, each sized by its share of the decision. Every finalist connects to every criterion exactly once, and that one connection carries both scores: the AI agent's, read from the CV, and the people's, from the interview or the work sample. Each comes with its evidence, its source and who gave it, and a note on whether the two agree.
The skills, experience, degrees, certificates and languages the agent pulled from the CVs are shared: there is one Python, held by all six finalists, and it points to the criterion it counts toward. Each finalist also has an overall evaluation with the weighted scores, the open concerns and the decision, which only a person sets. The weighted scores were calculated before import; the knowledge graph shows how they add up.
Four questions before the decision meeting
Where do the AI and the people disagree?
People have reviewed 36 of the 48 finalist-and-criterion scores so far. They agreed with the AI on 21, scored higher on 8 and lower on 7. Every gap wider than one point is on the same criterion: collaboration with analysts and stakeholders. Of the five finalists people have interviewed, only Candidate F got the same score from both.
It is the criterion a CV can't show. The agent gave Candidate C no score at all: the CV mentions mentoring without detail. In the interview, Candidate C described leading a joint incident review with support analysts that changed how alerts were routed, and scored 4. Candidate D went from 3 to 5, Candidate B from 4 to 2.
Is the top-ranked finalist the strongest?
Candidate B's CV reads like the job advert, and the agent ranked them first with the highest weighted score on the shortlist, 4.55. The completed human review lands at 3.50: people scored lower on six of the eight criteria, most sharply on collaboration, and the modelling exercise mixed two levels of detail in one table. The hiring manager put the candidate on hold.
Candidate D, second in the AI ranking at 3.80, is the only finalist whose completed human review came out higher than the AI's: 4.25, built on the best modelling work sample in the round. Candidate D is the one the hiring manager advanced.
Who misses a must-have, and is that the end?
Three criteria are must-haves. Candidate F misses one: professional English at C1. The candidate declares B2, and both the agent and the hiring manager scored the language below the bar. Everything else points the other way. AI and people agree on seven of the eight criteria, and Candidate F wrote the strongest pipeline code in the work sample.
The miss is flagged as an open concern, not used as a rejection. The decision stays pending while the hiring manager weighs language support against the technical strength. Candidate E's English is marked not assessed rather than met, because nobody has spoken with them yet.
Whose picture is still incomplete?
Three finalists have a complete human review, two a partial one and Candidate E none: every score on Candidate E is the agent's alone. A filter on the review status picks out the three still open in a click.
Candidate A shows why that matters. The agent ranked them fourth, but on every criterion people have scored so far, they rated the candidate the same as or higher than the AI. Two criteria worth 30% of the decision still wait for the work-sample panel, so the human weighted score stays at 0 and nobody compares a partial picture with complete ones.
How to read the knowledge graph
-
Start with the role
Click Data Engineer in the centre: a mid-level, hybrid role in the Data Platform team, and the story of why it was opened. The eight stars around it are the criteria. The biggest, Python and SQL for data pipelines, carries 20% of the decision.
-
Open a criterion
Click Production pipeline experience 3+ years. The detail panel shows its weight, 15%, that it is a must-have, and what a score of 1 to 5 means. Below that, all six finalists, each with the AI's and the people's score and the evidence behind them.
-
Open a finalist
Click Candidate D: AI rank 2, in the final round, one month's notice. The overall evaluation comes first, then the eight criteria, then what the CV showed, from Airflow, dbt and dimensional modelling to English C2 and an AWS certificate, each with where in the CV it was found.
-
Follow a shared skill
Click Apache Kafka. Three finalists hold it, Candidates B, C and F, and it counts toward Orchestration and streaming tools. Click through to that criterion to see whose Kafka experience held up in the work sample.
-
Read the notes at the side
The pink notes explain the graph itself. Rating scale, under the role, says what each score means and that 0 means not scored yet. What is left out and why lists what was kept out of the data, from names to graduation years. Human oversight and the law sums up the rules above.
Bring your own shortlist
You need three things you probably already have: the screening tool's export (rank, match score, extracted profile and per-criterion ratings with their reasons), the scorecard with weights and must-haves, and the interview and work-sample scores. Make each criterion and each skill one shared item, put the scores on the connection between a finalist and a criterion, and leave out names and anything that hints at age, gender or origin.
Your own AI assistant can turn those exports into one JSON file in this shape; paste it on the MindShare home screen and save. Update it as reviews come in. See how properties and types work, or ask us for help preparing the data.