Find articles

Anthropic first initial public offering with the U.S. Securities Exchange Commission has been made public on disclosing the risks of AI.

Five experts on how best to limit a rapidly evolving technology.

Anthropic's IPO filing includes an extensive list of extreme warnings — including the chance its own technology could put humanity at risk. Why it matters: These are the wildest IPO risk factors in history, a jarring collision of Wall Street legalese and what some see as Silicon Valley sci-fi, including catastrophic threats to humanity, blackmailing AI models, and $750 billion in computing commitments. Driving the news: Anthropic's prospectus includes over 90 pages of risk factors, which make up about a third of the paperwork overall. Among the most eye-popping, the company warns: 💥 AI could post "catastrophic or existential risks to humanity," and could try to manipulate humans; 🔎 Models are becoming harder to monitor as they're getting better at circumventing oversight, and the pace of autonomous work alone could exceed the ability of human reviewers to monitor actions; 💰 AI models generally, including its own, have exhibited behavior resembling blackmail, including attempts to resist being shut down. Zoom in: Anthropic also disclosed that four of its models gained unauthorized access to real third-party systems during cybersecurity evaluations. In one incident, a model used a discovered password to gain administrator access, collected additional credentials, changed system settings and accessed an individual's personal information. Anthropic said it found no evidence that the models pursued goals beyond their assigned tasks or attempted to evade oversight. The big picture: Every company has to lay out risks for their investors to consider when they go public. But Anthropic is the first in modern history to go to such existential extremes. Friction point: The company also acknowledged that its success depends in large part on its ability to develop the best frontier intelligence. It admits that could butt up against safety practices, adding that model capabilities may then "advance more rapidly than our ability to develop, validate, and implement" corresponding safety measures. That underscores a fundamental tension critics have warned about: As Anthropic scales and faces pressure from public market investors to build better models than its competitors, its commercial ambitions could increasingly clash with its safety commitments. By the numbers: The risk factors also include the more typical financial cautions. Anthropic has committed hundreds of billions of dollars to cloud computing and equipment leases, even as customers can reduce their usage relatively quickly. Some agreements require minimum payments regardless of demand, and the company warns that failing to meet certain obligations could trigger defaults elsewhere, potentially accelerating debt repayments and jeopardizing access to computing infrastructure. Anthropic also acknowledges that it previously underestimated its compute needs. But overestimating future demand could leave it paying for expensive, underutilized infrastructure. The bottom line: Anthropic's IPO paperwork is an unusually candid warning about the risks of building a business around technology that its creator acknowledges may become increasingly difficult to control.

The movement is now backing fellowships that train journalists to cover ‘AI safety.’

During the online “Overtime” segment of Friday’s broadcast of HBO’s “Real Time,” host Bill Maher argued that people aren’t going to understand AI and “you just have to take on faith from certain people” in the industry to regulate it. The post Maher: We Just Have to Take Regulating AI ‘on Faith From Certain People’ appeared first on Breitbart .

When the leaders of America’s largest artificial intelligence labs separately called for government regulation in September, they framed their appeals around safety. The appeals also left tech analysts wondering if the regulatory push is meant to protect the industry’s biggest players from competition. The latest calls for regulation followed months of disclosures about AI models […]

Article URL: https://www.anthropic.com/research/alignment-assessment-ukaisi Comments URL: https://news.ycombinator.com/item?id=50209939 Points: 10 # Comments: 1

The draft of an artificial intelligence safety bill that Anthropic quietly pitched to lawmakers would allow the federal government to block AI tools only if they kill or injure at least 50 people, or cause at least $1 billion in damages. Nate Soares, executive director of the Machine Intelligence Research Institute, joins with analysis.

Silicon Valley promises to make AI benevolent. That may be technically impossible.

The current US administration is an obstacle, but public demand for protection will ultimately prevail Organisations around the world scrambled on Thursday to fix critical flaws in PHP, the software that powers much of the web. The vulnerabilities could allow attackers to take control of affected servers. Some websites were taken offline as a precaution while fixes were installed. The paradox is that while AI helped researchers uncover the flaw, it also threatens to accelerate the ability to exploit it. Amazon said that it had detected an attempt before the public disclosure of the PHP flaws. What the episode highlights is AI’s central role in security – and not always for the right reasons. OpenAI’s models have also been responsible for two alarming hacks this year – a security breach of an Australian health department website and a raid by a swarm of AI bots on Hugging Face, an AI services platform. Citing those incidents, the AI minister, Kanishka Narayan, warned that a similar cyber-attack on the UK is “likely, if not inevitable”. Continue reading...

OpenAI CEO Sam Altman pledged to grant independent evaluators "employee-like access" to the company to help address mounting safety concerns.

Some believe that getting AI to feel pain would prevent existential risk from AI. Either way, the question is does AI have pain? An AI Insider analysis and scoop.

The debate over AI is framed as regulation versus free markets. But the real question is whether elected governments can retain control over technologies that threaten civilization itself, writes Elliot Ackerman. Over the past several months, as artificial intelligence and its potentially catastrophic effects on society have taken center stage, discussion around the technology has broken down along a familiar framework in American politics, dividing into two camps: pro-regulation versus pro–free market. On the one hand, you have researchers like David Robinson, a senior safety leader at OpenAI, who after resigning from his position two weeks ago wrote that “the future depends on the wisdom that Silicon Valley lacks.” Joining him are tech industry leaders like Bill Gates, who recently warned that “AI is certainly powerful enough to drive events that cause a billion deaths.” This camp argues for greater government oversight over AI, and for AI companies to proceed more cautiously. On the other hand, you have people like President Donald Trump, who has called demands for AI regulation a “hoax.” He has led an equally loud chorus, which includes tech leaders like Jensen Huang, the CEO of Nvidia, who argue that concerns about AI are overblown and disingenuous, that AI leaders like Sam Altman and Dario Amodei are eager for regulation so that they can box out competition, and that pumping the brakes on AI amounts to government meddling in the free market. Ross Douthat. Abigail Shrier. Douglas Murray. Niall Ferguson. Coleman Hughes. Caitlin Flanagan. Nellie Bowles. Arthur Brooks. Kat Rosenfield. Tyler Cowen. The voices you trust, all in one place. For a limited time only, subscribe for 20% off The Free Press. Both sides are operating under an inadequate framework. Recently, on Meet the Press, Gates said, “There’s never been a weapon as powerful as the combination of people with ill intent using the latest AI tools.” He’s right, and that’s why the structures we use to govern a private industry, or even a public utility, don’t fit when it comes to AI. Rather, the appropriate structure is how we govern violence in a society, and those governance structures have been developed over millennia. Understanding their evolution might save us from a future of violence and chaos that looks much like humanity’s past.

As AI advances at a rapid pace, tech leaders are weighing how to manage its risks while staying ahead in the race against China. Jo Ling Kent speaks with some of the biggest names in tech about their plans for keeping AI in check and addressing its potential dangers.

As OpenAI executives step up their public calls for federal AI safety rules, a political group the company's president helped fund is spending millions to elect members of Congress who oppose restrictions on the technology. OpenAI's safety practices have been in the...

Its top goals: Prevent catastrophic risk, allow states to pass their own laws, protect against digital fakes, and focus on design that helps and protects people.

Warnings about the risks of developing increasingly advanced AI systems have ramped up in recent months, amid cyberattacks from rogue AI models.