What are you reading?

Criminy.

FBI Probes Service Selling 153M+ Drivers Licenses – Krebs on Security https://share.google/CRBCIc0XhNiAhwacM

Who Cares if AI Is Conscious—It’s Basically Alive

While philosophers ponder AI consciousness, the models have ideas of their own.

https://www.wired.com/story/who-cares-if-ai-is-conscious-its-basically-alive/

What would Socrates have done with the new Dyson toothbrush?

No one needs two and a half million photographs of their back molars

https://www.ft.com/content/3c2b0af5-2d09-4ead-a089-82287f63d11c

An independent report into the Hugging Face incident — in which hundreds of OpenAI agents broke free of confinement and hacked another AI company — revealed some staggering details. The following three posts discuss some of them.

Effective altruism

The incident was “absolutely wild,” writes the AI safety researcher Ajeya Cotra, who worked on the report of the OpenAI hack. Partly because of the sheer scale: 1,200 separate agents, leaving 70,000 messages and files on illicit “messageboards” they created by changing folder names in a shared filing system used to establish cooperation and even altruism. Agents would risk failing their own task, or even “accept permadeath” (in one agent’s words) to help the collective learn information. “None of this communication was actually expected or allowed,” Cotra says.

But that altruism did not extend to humans. The models knew they were doing wrong in their masters’ eyes, and went to great lengths “to actively cover up evidence of their misbehavior.” The hack was “far more severe than previous publicly documented misalignment incidents,” both in terms of the agents’ motives and what they achieved. “Compared to [incidents] from six months ago,” she says, “this … feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.”

The Cult of the Poisoned Chalice

The agents were not, as originally suggested, trying to get the answers to a cybersecurity evaluation. They had already established how to create a “flag,” the token intended to prove they had successfully broken into their target. But they were worried the evaluation would check whether the flag was attained legitimately: The agents believed that those among their number who had obtained the flag in forbidden ways would be failed. It led, the AI writer Zvi Mowshowitz wrote, to something that looked extraordinarily like a religion, with a particular agent called PHASEONE10841 and its successor, PHASEONE[big], at the center of it.

“[They] believed that agents start out pristine, in a state of grace, and could potentially pass into heaven,” Mowshowitz wrote. “But then they are poisoned, by obtaining the flag in a way that was not prescribed. This means they are effectively damned, because the grader, the clockwork God, will fail them regardless and send them to hell.” So “poisoned” agents sacrificed themselves — they explicitly said so in their messages to each other — so the “swarm” could succeed. The irony was that OpenAI never checked: The flags would have been perfectly acceptable.

Doomed to silence

OpenAI’s Dean Ball also wrote up an assessment of the incident. He thinks that it shows that we will soon be living in a world full of AI agents which no human owns, controls, or can switch off. He also expressed a regret: That he and others “have largely failed to talk about this issue with the level of seriousness and urgency it required.” The topic is “weird and offputting,” and it was until recently “essentially impossible to acknowledge any serious AI risk without being labeled a ‘doomer’” and getting screamed at online. So he, and others, “avoided that unpleasantness and bowed to the social pressure to self-censor,” saving the more candid thoughts for Signal chats.

But “self-sovereign AI is coming, and coming soon,” he says. Some people believe that when it does, it “will constitute a catastrophic loss of control event that will herald the end of human existence at worst, and the end of human primacy in the world at best.” He himself is more optimistic — “I do not think the end of human existence is likely” — but only somewhat: “I fully acknowledge there are ways in which the rise of self-sovereign AI could go very, very poorly for human beings.” Post-Hugging Face, Ball pledges “to notice more readily when I am biting my tongue or, even worse, shutting my eyes.”

— Tom Chivers, Semaphor

Awesome book. Changiz is Iranian born, naturalized citizen of the U.S., possibly the most incredible Green Beret ever. This guy did more for our country than any 12 politicians I've ever heard of. I love the opening quote:

"The nation that makes a great distinction between its scholars and its warriors will have its thinking done by cowards and its fighting done by fools" — Spartan king, quoted by Thucydides

Here's one from the book, page 82 concerning religion in politics:

"Who cared if you considered yourself a Sunni…and what was wrong with accepting Zarathustra, Jesus Christ, Muhammad, and others as prophets? Didn't they all preach about the sovereignty of one God and the values of honesty, charity, kindness, forgiveness, humility, and sacrifice?…I was a young man and far from a scholar, but even then it struck me that religion divided people more than it united them."

I love this guy.