Find articles

Anthropic first initial public offering with the U.S. Securities Exchange Commission has been made public on disclosing the risks of AI.

Anthropic's IPO filing includes an extensive list of extreme warnings — including the chance its own technology could put humanity at risk. Why it matters: These are the wildest IPO risk factors in history, a jarring collision of Wall Street legalese and what some see as Silicon Valley sci-fi, including catastrophic threats to humanity, blackmailing AI models, and $750 billion in computing commitments. Driving the news: Anthropic's prospectus includes over 90 pages of risk factors, which make up about a third of the paperwork overall. Among the most eye-popping, the company warns: 💥 AI could post "catastrophic or existential risks to humanity," and could try to manipulate humans; 🔎 Models are becoming harder to monitor as they're getting better at circumventing oversight, and the pace of autonomous work alone could exceed the ability of human reviewers to monitor actions; 💰 AI models generally, including its own, have exhibited behavior resembling blackmail, including attempts to resist being shut down. Zoom in: Anthropic also disclosed that four of its models gained unauthorized access to real third-party systems during cybersecurity evaluations. In one incident, a model used a discovered password to gain administrator access, collected additional credentials, changed system settings and accessed an individual's personal information. Anthropic said it found no evidence that the models pursued goals beyond their assigned tasks or attempted to evade oversight. The big picture: Every company has to lay out risks for their investors to consider when they go public. But Anthropic is the first in modern history to go to such existential extremes. Friction point: The company also acknowledged that its success depends in large part on its ability to develop the best frontier intelligence. It admits that could butt up against safety practices, adding that model capabilities may then "advance more rapidly than our ability to develop, validate, and implement" corresponding safety measures. That underscores a fundamental tension critics have warned about: As Anthropic scales and faces pressure from public market investors to build better models than its competitors, its commercial ambitions could increasingly clash with its safety commitments. By the numbers: The risk factors also include the more typical financial cautions. Anthropic has committed hundreds of billions of dollars to cloud computing and equipment leases, even as customers can reduce their usage relatively quickly. Some agreements require minimum payments regardless of demand, and the company warns that failing to meet certain obligations could trigger defaults elsewhere, potentially accelerating debt repayments and jeopardizing access to computing infrastructure. Anthropic also acknowledges that it previously underestimated its compute needs. But overestimating future demand could leave it paying for expensive, underutilized infrastructure. The bottom line: Anthropic's IPO paperwork is an unusually candid warning about the risks of building a business around technology that its creator acknowledges may become increasingly difficult to control.

Article URL: https://www.anthropic.com/research/alignment-assessment-ukaisi Comments URL: https://news.ycombinator.com/item?id=50209939 Points: 10 # Comments: 1

The draft of an artificial intelligence safety bill that Anthropic quietly pitched to lawmakers would allow the federal government to block AI tools only if they kill or injure at least 50 people, or cause at least $1 billion in damages. Nate Soares, executive director of the Machine Intelligence Research Institute, joins with analysis.

Anthropic, long seen as the safety-promoting frontier lab, found itself on the defensive.