Find articles

Article URL: https://www.anthropic.com/research/alignment-assessment-ukaisi Comments URL: https://news.ycombinator.com/item?id=50209939 Points: 10 # Comments: 1

In the race to fend off AI-enabled cyber threats , security teams are confronting a new problem: The AI models they rely on to investigate attacks can suddenly refuse to help. Why it matters: While AI giants race to build AI tools that can respond quickly to hacking or other malicious online threats, strict guardrails on their most powerful models lock out defenders mid-investigation, leaving them scrambling for alternatives at critical moments. What's happening: OpenCode, an open-source AI coding agent, said it was compromised over the weekend, allowing attackers to obtain API keys belonging to some customers, including keys for OpenAI, Anthropic and Google services. OpenCode said the breach was "related to a large scale distillation attack that impacted 1.4% of OpenCode Console users." OpenCode co-creator Dax Raad said attackers had been targeting its systems continuously for 48 hours, forcing the company to keep its own AI agents running around the clock to respond. OpenAI , Google and Anthropic have said they worked with affected customers to revoke exposed API keys and address unauthorized usage charges. Zoom in: Raad said on X that frontier AI models initially helped investigate the breach but then "started refusing." The models' safety restrictions reportedly began blocking requests as the investigation turned to potentially malicious code, even though OpenCode was trying to defend its own systems. OpenCode then gained access to OpenAI's Daybreak Blue program, which gives vetted cybersecurity defenders access to models with fewer restrictions. "We definitely need to be more proactive and we've made so many improvements over the last few days," Raad wrote on X Monday. The big picture: The incident underscored a dilemma facing cybersecurity teams trying to figure out when to use powerful proprietary AI models, which come with built-in restrictions, or open-weight alternatives that offer greater control over how they're deployed. When Hugging Face discovered suspicious activity on its systems that it later linked to OpenAI's own agents, its security team turned to a Chinese open-weight model to help investigate after encountering restrictions on frontier models. Yes, but: AI companies maintain that they need to gate access to their most powerful models to ensure that malicious actors can't abuse their cyber capabilities. Between the lines: Cybersecurity teams will have to figure out how to balance the need for faster AI-powered defenses with restrictions designed to keep those same capabilities out of attackers' hands, Chris Jones, chief technology officer at Nightwing, told Axios. "We're all going to have to figure out how that race gets modulated over time," Jones said. "That's a big piece here." Go deeper : Anthropic and OpenAI are now cybersecurity's kingmakers