AI companies are competing for talent with more than pay packages and GPU access. Their position on AI safety may be starting to matter too

Hello and welcome to Eye on AI.

The leadership shakeup at top AI companies continued this week. Jan Leike, one of OpenAI’s lead safety researchers who resigned earlier this month stating that safety at the company “has taken a backseat to shiny products,” joined rival firm Anthropic.

“I'm excited to join @AnthropicAI to continue the superalignment mission! My new team will work on scalable oversight, weak-to-strong generalization, and automated alignment research,” Leike posted on X.

The new role sounds almost identical to the one Leike had at OpenAI, where he co-led the “superalignment” team tasked with understanding the risks of future powerful AI systems and ensuring they could be controlled. The OpenAI superalignment team—which Leike ran alongside Ilya Sutskever, OpenAI cofounder and chief science officer who also resigned last month—lasted less than a year before its co-leads stepped down and OpenAI disbanded the effort. Having your top AI Safety folks all jump ship is a bad look, to say the least. The company once positioned the team as vital, but OpenAI never gave the team the resources it promised them in order to do the job, as my Eye on AI coauthor Jeremy Kahn exclusively reported in Fortune.

Leike’s safety-related departure comes alongside a whirlwind of recent scandals that have chipped away at public trust in OpenAI—from the intense non-disparagement agreements the company made employees sign to using a voice that sounds eerily similar to Scarlett Johansson’s after she denied the company’s offer to voice ChatGPT. (OpenAI has denied it deliberately tried to imitate Johansson’s voice and has shown the Washington Post documents to back up its position.)

Former OpenAI board member Helen Toner this week also broke her silence about why the board abruptly fired Sam Altman last November, saying he didn’t notify the board about ChatGPT’s release and gave inaccurate information about the company’s safety processes “on multiple occasions,” among other reasons. (Current OpenAI board members Bret Taylor and Larry Summers responded to an editorial that Toner and her fellow former OpenAI board member Tasha McCauley wrote in The Economist calling for regulation of AI companies with an editorial of their own. Summers and Taylor said that in six months of near-daily contact with OpenAI, they had found Altman to be “highly forthcoming on all relevant issues and consistently collegial with his management team.” They also noted that Altman and OpenAI had repeatedly called for governments to regulate AI and AI companies.)

So far, discussions around incentives in the intensifying war for top AI talent have mostly revolved around multimillion-dollar compensation packages, and the ability of Big Tech companies such as Meta to offer AI experts access to thousands of GPUs, the specialized computer chips needed to build and run AI models. With their skills in such high demand, experienced AI researchers and engineers have reportedly fetched pay packages as high as $10 million. Lekie’s jump to rival Anthropic to do basically the same job, however, suggests that AI companies’ demonstrated commitment to safety may become an increasingly important factor in the AI talent war. There are a lot of folks who want to see AI developed and deployed responsibly, and the stream of resignations from OpenAI over safety concerns shows that top talent will go elsewhere if they feel the company they’re working for is on a dangerous path.

The stakes are getting higher by the day as AI (and its harmful effects) advance rapidly, with insufficient guardrails. For AI professionals, whether they are concerned with existing harms, like deepfakes, or long-term risks, like AI destroying humanity, as Leike is, there’s no time to waste at a company that won’t take their work or the risks seriously. Engineers who are developing AI systems might also think twice about where they bring their talents if they believe certain companies are acting recklessly, fearing they’ll be blamed for unleashing harmful AI and all the consequences that come with it.

So far, Anthropic has been the main beneficiary of this. The company has largely positioned itself as more safety-focused than OpenAI, and Leike isn’t the first leader to defect from OpenAI to Anthropic over safety concerns. In fact, Anthropic was founded by two OpenAI executives who left the firm in 2020 over concerns it was prioritizing commercializing the technology over safety.

Since its launch, Anthropic has continued to take steps that point toward more conscientious development of AI. The company’s “constitutional AI” research, for example, involved polling American adults about what sort of principles they think would be important for an AI model to abide by and then training a smaller version of Claude—the company’s ChatGPT competitor—based on their suggestions. Last week, Anthropic published a research paper detailing its latest efforts to understand the inner workings of large language models. And while OpenAI has moved away from its nonprofit beginnings and seen its board splinter (and even oust CEO Sam Altman) over safety concerns, Anthropic has taken sharp moves in the other direction. Last year, Anthropic registered as a Long-Term Benefit Trust (LTBT), which bakes in a governance mechanism that allows an independent body of financially disinterested members to dismiss board members if they fail to uphold the company’s stated mission to “responsibly develop and maintain advanced AI for the long-term benefit of humanity.” Though to be fair, it’s not clear exactly how this process would play out. A profile of the company published today in Time further dives into Antrhopic’s approach and hope to create “a race to the top” on safety.

Besides rival AI companies, the lure of doing right by humanity could also boost another entity in its AI recruiting: the federal government. With the urgent need for AI expertise across government agencies—including ones that will impact how AI is deployed—AI professionals who have felt burned by the tech industry’s pursuits for endless profits might come to believe they could make a stronger impact in the public sector. Seeing as the government is unable to compete with sky-high private sector salaries, it may be its best sales pitch—and some government officials have made it, though maybe not in so many words. Of course, without passing measures to regulate AI, and by continuing to give companies like OpenAI free reign, the government’s message that it’s a place to ensure AI is developed safely may start to ring hollow.

And with that, here’s more AI news.

Sage Lazzaro
sage.lazzaro@consultant.fortune.com
sagelazzaro.com

This story was originally featured on Fortune.com