Almost every engineering team has hired the person who breezed through the interview and then couldn't ship. Most have also passed on someone who would have been great because they froze at a whiteboard. The same interview formats keep producing both mistakes, and the industry keeps trusting them anyway.
Here is the pattern worth noticing. The interviews that tell you the least put a candidate somewhere that looks nothing like the actual job: alone, silent, nothing to look up, no colleague to ask, a stranger watching and grading. The interviews that tell you the most recreate the job and watch someone do it.
The interviews that predict nothing drop a candidate somewhere that looks nothing like the job. The ones that predict performance recreate it.
Start with the failure you actually fear on a real team. It isn't the developer who can't invert a binary tree on command. It's the one who gets stuck on a Tuesday and says nothing until Thursday. Two days gone because asking felt like admitting weakness. No algorithm puzzle will ever find that person. The right interview will.
The only question that matters
Strip away the format wars and one question is left: does your interview predict how someone performs once they are hired? Not how confident you felt afterward. Not whether they came across as smart. Whether the people who score well go on to do the work well.
Almost no team can answer that about its own process. We run the same loop for years, argue about it in debriefs, and never once check the scores against how those hires actually turned out. So before defending any format, ours included, it is worth asking whether you know that yours predicts anything at all.
What the evidence supports, and what it doesn't
One belief needs clearing out first, because it sits underneath a lot of bad interviews: the idea that the smartest person in the room is the safest hire. For years, hiring advice leaned on research suggesting that raw cognitive ability predicted job performance better than almost anything else (Schmidt & Hunter, 1998). Those estimates have been revised downward considerably in the last few years: a major reanalysis found that the old estimates were inflated (Sackett et al., 2022), and newer data has pushed the cognitive-ability figure lower still (Sackett et al., 2024). Structured, job-relevant assessment now looks like the stronger bet. If your loop is built around proving how clever a candidate is under pressure, the ground under it has moved.
What the research does support is worth stating plainly. Using consistent questions and anchored scoring improves how well interviews predict performance and clearly beats leaving them unstructured (McDaniel et al., 1994; Levashina et al., 2014). Giving candidates an actual sample of the work also predicts later job performance (Roth et al., 2005). And one controlled study of technical interviews found performance fell by more than half in a traditional observed, think-aloud whiteboard condition compared with a private condition (Behroozi et al., 2020).
Two limits are worth naming, though. The research backs sampling the actual work and structuring how you score it, but it has not named a single best format for software engineers, and it has not certified ours. That our particular loop is the right way to do it is our judgment, not a finding lifted from a study.
The format scorecard
Most format debates go in circles because people compare the wrong thing: which format feels rigorous, which one is kindest, which one they happen to be used to. Here is the comparison that matters: what each format plausibly measures and whether there is any reason to believe it predicts the job.
| Format | What it plausibly measures | Reason to think it predicts the job | Our verdict |
|---|---|---|---|
| Unstructured "culture fit" chat | Rapport and presence, mixed with how similar the candidate feels to you | Some signal, but weaker and more bias-prone than structured formats (McDaniel et al., 1994; Levashina et al., 2014) | Qualified.Keep the read, but anchor it to behavior, not gut |
| LeetCode and algorithm puzzles | Algorithmic recall, interview prep, coding against a clock | Little direct evidence it tracks day-to-day engineering | Negative.Drop it. It buys confidence you have not earned |
| Take-home project | Real coding on a realistic task, done alone | Supported in principle: work samples predict later performance (Roth et al., 2005) | Qualified.Useful, but it amputates the collaboration |
| Observed whiteboard | Stress tolerance at least as much as problem-solving | Performance fell sharply in an observed, think-aloud whiteboard condition compared with a private condition (Behroozi et al., 2020) | Negative.Retire it |
| Collaborative live session, full tools | Competence in context, plus how someone collaborates | Inference from the work-sample and structure research | Positive.What we run |
A few rows deserve a word.
LeetCode is the one people defend hardest, so be precise about the objection. Algorithm puzzles clearly measure something: reasoning, preparation, the ability to write code while a clock runs. What is hard to find is direct evidence that the something predicts the work we hire for. If your engineers rarely invert trees from memory, alone, unable to look anything up, the burden is on you to explain why that is the right proxy. A realistic problem tests competence directly, in a setting closer to where it counts. For the work we hire for, we would choose collaboration over LeetCode every time.
The observed whiteboard is worse than its reputation. In one controlled study, performance fell by more than half in a traditional observed, think-aloud whiteboard condition compared with a private condition. Two honest caveats matter: the observation and the think-aloud demand were bundled together, so you cannot pin the effect on one alone, and both groups worked without their normal tools, so the study does not prove that missing tools were the cause. The finding that survives is still damning enough. The traditional observed format can make people substantially worse at the task.
The "let's just chat and see if they fit" round is the part of a loop most exposed to bias, because "we hire for collaboration" is one short step from "we hire people who remind us of ourselves." That does not make the looser conversation worthless; an experienced interviewer reads real signal in how present, curious, and engaged someone is, and the job of structure is not to kill that instinct but to keep it honest by making you name what the candidate did rather than settle for the fact that you liked them. So keep the conversation. Just score the behavior it surfaces, not the vibe.
What we would run
Here is ours, with a clear line between where the evidence stops and where our judgment starts. The evidence gives constraints: structure the evaluation, make the task resemble the job, and watch behavior that matters on the actual job. Our version of that is a collaborative live session on a realistic problem, with every tool an engineer has on a normal day. They can search. They can use AI. They can ask the interviewing lead a question. The lead behaves like a colleague rather than a proctor and guides instead of handing over the answer.
Competence is still a gate, and a hard one. The problem is real, and a candidate who cannot reason through it does not pass just because they were pleasant company. But once someone clears that bar, competence is not the thing we choose on. Collaboration is. Concretely, we are watching four things:
- Do they ask good questions, or guess in silence?
- Do they take a hint and run with it, or let it drop?
- Can they disagree with the lead well, holding a position without digging in past the evidence?
- Do they know when to stop and ask versus when to push on alone?
And here is the part that is ours, offered as conviction rather than as a research finding. When we hand a candidate the freedom to ask for help and they refuse to use it, that counts against them. Struggling in silence with a search bar one tab away is not grit. It is the precise failure that costs a real team two days and a slipped deadline: the engineer who will not say "I'm stuck." We would rather hire the person who asks a slightly embarrassing question in the first hour than the one who suffers quietly until Thursday. No puzzle surfaces that trait. This interview does.
We would rather hire the person who asks the slightly embarrassing question in hour one than the one who suffers in silence until Thursday.
Keeping it honest
The fastest way to wreck all of this is to run it on vibes, so a few guardrails carry real weight.
Score before you talk. Every interviewer writes down their rubric scores before the debrief starts. Otherwise "collaboration" becomes the word people reach for to justify a candidate they already liked, and the bias you were trying to remove strolls back in through the side door.
The lead guides and does not solve. This does make the session partly subjective, since some leads answer questions better than others. What you standardize is the task, the rubric, and the boundaries for assistance. Within those boundaries, the lead guides rather than solves. That takes trained interviewers, which most shops skip and we do not.
No format escapes stress, and we do not pretend ours does. Anything graded carries pressure, take-homes very much included. What we removed are the artificial stressors that have nothing to do with the work: the silent audience, the ban on looking things up. What is left is the ordinary stress of being evaluated, and every format shares that.
Know what an hour cannot tell you. One session gives you real evidence about how a person reasons, collaborates, and handles their tools. It tells you almost nothing about whether they will still be reliable in month six. That is fine, as long as you do not pretend otherwise. Follow-through is what references, a paid trial, and the first ninety days are for.
On AI, briefly, because it deserves its own post. Working with AI is part of many engineering workflows now, so interview for it the way the job actually uses it. The honest exception is regulated or IP-sensitive shops that restrict AI for good reasons; if the job forbids it, the interview should too. Everywhere else, watching how someone works with AI, what they trust it for, when they stop to check it, how they steer it, is one of the more revealing things you can observe in an hour. More on that another time.
Structure is what turns collaboration from a feeling you have about a candidate into something you can actually measure.
Stop optimizing the format
If there is one thing to take from this, it is to stop tuning the format and start recreating the job. Pick the interview that looks most like the work, make the task real, let people use what they would really use, and score the thing you claim to care about instead of the thing that is easy to grade.
Then do what almost nobody does. Go back through your last two years of hires and ask whether the people who scored well are the ones doing well now. If you have never looked, you do not have an interview process. You have a ritual you happen to trust.
So we will ask it straight, and we would welcome the argument. Which format will you defend, and have you ever once tested it against how those hires actually turned out?
The research
The claims above rest on a handful of sources. A note on reading them: the estimates in this literature have been revised more than once, and remain debated, so the honest takeaway is the direction of travel, not any single decimal.
- Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274. The classic synthesis that anchored decades of hiring advice, and the figures later work revised.
- McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79(4), 599–616. Structured interviews outpredict unstructured ones.
- Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037. Samples of the actual work predict job performance.
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(11), 2040–2068. Corrected long-standing overestimates; cognitive-ability validity came down, and structured interviews moved to the top of the list.
- Sackett, P. R., Demeke, S., Bazian, I. M., Griebie, A. M., Priest, R., & Kuncel, N. R. (2024). A contemporary look at the relationship between general cognitive ability and job performance. Journal of Applied Psychology, 109(5), 687–713. Twenty-first-century data put the cognitive-ability estimate lower still.
- Levashina, J., Hartwell, C. J., Morgeson, F. P., & Campion, M. A. (2014). The structured employment interview: A narrative and quantitative review of the research literature. Personnel Psychology, 67(1), 241–293. Structure raises validity and reduces susceptibility to bias.
- Behroozi, M., Shirolkar, S., Barik, T., & Parnin, C. (2020). Does stress impact technical interview performance?. Proceedings of ESEC/FSE 2020. In a randomized trial, participants in a traditional observed, think-aloud whiteboard condition performed substantially worse than participants solving privately. Small sample, and observation was bundled with a think-aloud demand, so read it as a strong signal rather than a settled estimate of the effect of observation alone.