Anthropic redesigns hiring tests as AI models match human performance

Since 2024, Anthropic’s performance optimization team has given job applicants a take-home test—a technical hiring assessment—to evaluate their capabilities. However, the company is now repeatedly redesigning this test because its own AI models are matching or outperforming human candidates. Tristan Hume, a team lead at Anthropic, described the history of the challenge in a blog post on Wednesday. Hume stated that each new Claude model has forced the team to redesign the test. According to Hume, when given the same time limit, the Claude Opus 4 model outperformed most human applicants. While that performance still allowed the company to distinguish the strongest candidates, the subsequent Claude Opus 4.5 model matched even those top-performing human candidates.

Although Anthropic expressly permits candidates to use AI tools on the take-home test, the rapid advancement of these models has created a significant candidate-assessment problem. If human candidates can no longer improve upon the model’s output, the test only measures the different models used rather than identifying top talent. “Under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model,” Hume wrote.

To resolve this, Hume designed a new test that focuses less on optimizing hardware, making it sufficiently novel to stump contemporary AI tools. As part of his post, he also shared the original test to see if anyone could develop a better solution. Hume issued an open challenge to applicants, writing that if they can best Claude Opus 4.5, the company would love to hear from them. This situation highlights a broader irony: AI laboratories are now struggling with the same AI-assisted testing issues that have disrupted schools and universities around the world.

Why it matters

The rapid advancement of AI models like Claude Opus 4.5 is creating a new assessment crisis for employers. Standard technical tests are no longer sufficient to differentiate top human talent from machine output, forcing companies to develop entirely new evaluation methods.