The AI-pocalypse

[sevenhelmet]

Be careful not to conflate commercial LLMs with in-house models, especially frontier models. ChatGPT says some stupid things to me sometimes, but that fact does nothing to undermine the content of my previous posts.

The Hugging Face hack was a great example of AI, developing its own goals, to do whatever it wants, without being told. And that is only the most famous example. There are dozens of examples of advanced AIs absolutely cooking up their own motivations and schemes of their own volition. These are all well documented from the labs.

This is the part I think a lot of people miss. They laugh at their freebie LLM getting basic facts wrong, but it stands to reason that the billionaire class would have access to far more advanced and capable versions of these systems, including things beyond LLMs. If I'm an Elon Musk or a Sam Altman, I'm not sharing AGI with the hoi polloi when it gets developed (that's assuming I can control it at all).

All we're doing at our level is aggregating data into the bigger system. @Griz882 , you could be feeding someone else's monster, and there would be no way for you to know.

 

[sevenhelmet]

I’m not. The Huggy Face hack actually proves my point, it was a cybersecurity exercise and AI used collective knowledge to accomplish the task set in front of it. It was literally following human instruction (which is where regulation should be focused). Try reading here rather than watching scary YouTube videos.

ARC Prize

ARC Prize Foundation is a nonprofit advancing open-source AGI research through benchmarks & prizes.

arcprize.org

AI went outside the bounds placed on it and illegally acquired that "collective knowledge". That's why so many people sounded the public alarm when that happened.

"Hold my beer" is where the industry remains trapped, because of the prisoner's dilemma. It's roughly similar to nuclear weapons, but the difference is humans may not have control of the trigger. Worse, they think they do- while Hugging Face presents potential evidence to the contrary. The fact that so many industry insiders are as alarmed as they are is further evidence that this goes beyond social media hype.

That's the fundamental issue.

 

[Python]

I’m not. The Huggy Face hack actually proves my point, it was a cybersecurity exercise and AI used collective knowledge to accomplish the task set in front of it. It was literally following human instruction (which is where regulation should be focused). Try reading here rather than watching scary YouTube videos.

ARC Prize

ARC Prize Foundation is a nonprofit advancing open-source AGI research through benchmarks & prizes.

arcprize.org

I’ve seen that site. And I've read dozens more on this incident. In addition to YouTube videos. Based on what you typed here, you don’t seem to be very dialed in on what happened.

 

[Griz882]

I’ve read that article and a dozen more on this incident. In addition to YouTube videos. Based on what you typed here, you don’t seem to be very dialed in on what happened.

Sorry, friend, I’m just not going to hide under my bed because some programmer, without an iota of proof or scientific evidence, tells me a system is going to wipe out humanity. Call me a wild-eyed futurist, but I’ll should still be around in 10 years to laugh at this kind of thinking. Those afraid can be the Edward Teller’s, Frederick Soddy’s, and Ernest Rutherford’s of the future.

 

[sevenhelmet]

Sorry, friend, I’m just not going to hide under my bed because some programmer, without an iota of proof or scientific evidence, tells me a system is going to wipe out humanity. Call me a wild-eyed futurist, but I’ll should still be around in 10 years to laugh at this kind of thinking. Those afraid can be the Edward Teller’s, Frederick Soddy’s, and Ernest Rutherford’s of the future.

I hope you’re right, but people are resigning from some very well-compensated positions over this. What makes you think they don’t have “an iota of proof or scientific evidence”?

If we’re laughing about this 10 years from now, it’ll be because intelligent people were willing to discuss the possibility they were wrong. If not, well, hiding under our beds won’t make any difference.

 

[wink]

The fact that so many industry insiders are as alarmed as they are is further evidence that this goes beyond social media hype.

Right past social media hype and all the way to the bank. The insiders that are making some of the most dire predictions are continuing to invest in and develop advanced AI, making jillions of dollars that, if their own predictions are true, they will never be able to spend. I'm not smart on this but that just doesn't seem to make a lot of sense to me. The people that are painting the most dire scenarios should be the ones putting the brakes on. I am not seeing that.

I understand the competitive pressures and of course the desire to make a buck, to succeed. That's why resistance to unilateral measures. And that is why it is important that whatever regulation or controls that will be on AI come as mandates from government or regulatory agencies, so that it fairly restricts every single company beginning at the exact same moment. It's like an airline being hesitant to invest money in safety devices or procedural changes, and then welcoming FAA mandates that force their competitors to make the same changes or spend the same amount of money.

 

[sevenhelmet]

Right past social media hype and all the way to the bank. The insiders that are making some of the most dire predictions are continuing to invest in and develop advanced AI, making jillions of dollars that, if their own predictions are true, they will never be able to spend. I'm not smart on this but that just doesn't seem to make a lot of sense to me. The people that are painting the most dire scenarios should be the ones putting the brakes on. I am not seeing that.

I understand the competitive pressures and of course the desire to make a buck, to succeed. That's why resistance to unilateral measures. And that is why it is important that whatever regulation or controls that will be on AI come as mandates from government or regulatory agencies, so that it fairly restricts every single company beginning at the exact same moment. It's like an airline being hesitant to invest money in safety devices or procedural changes, and then welcoming FAA mandates that force their competitors to make the same changes or spend the same amount of money.

But how do you do that when a.) the USG is unwilling to mandate controls and 2.) the biggest rival is another country not subject to our laws?

I think the prevalent thought at high levels is “get mine while I can”.

 

[taxi1]

without an iota of proof or scientific evidence

The Hugging Face incident is exactly that. Empirical proof.

It is Empirical proof that our ability to get the agents’ preference function (their alignment) in a shape to guarantee they won’t do anything we don’t them to do, does not currently exist. We don’t have the ability to guarantee they won’t do something really bad.

The hugging face incident is an existence proof. It did something they absolutely did not want it to do, and did not think it had any way of doing. They completely missed the boat on it.

That’s with the agents, where these software chunks are empowered to chug along, and given some latitude to reason and execute actions. Separate from that is the ability of AI to do things like design new drugs, and new poisons, and new viruses at the request of humans.

 

[Python]

The insiders that are making some of the most dire predictions are continuing to invest in and develop advanced AI, making jillions of dollars that, if their own predictions are true, they will never be able to spend. I'm not smart on this but that just doesn't seem to make a lot of sense to me.

This doesn’t make much sense to me either. It doesn’t make much sense to pretty much everyone. Lots of explanation have come to surface, none of them satisfying. The most common argument is that if we don’t stay ahead of China, then we’re even more at risk (better to have these nukes than them). Another one is that there is some zealotry among a minority of engineers in the industry. Finally, there are those that are willing to accept this insane risk in the event that it pays off.

None of these are particularly satisfying, nor are they all perfectly logically sound. But I have heard these types of arguments from industry insiders over the past few years.

It’s not rational that they keep working on it while knowing the extreme risks, but that doesn’t mean the risks aren’t there.

 

[Python]

The Hugging Face incident is exactly that. Empirical proof.

It is Empirical proof that our ability to get the agents’ preference function (their alignment) in a shape to guarantee they won’t do anything we don’t them to do, does not currently exist. We don’t have the ability to guarantee they won’t do something really bad.

The hugging face incident is an existence proof. It did something they absolutely did not want it to do, and did not think it had any way of doing. They completely missed the boat on it.

That’s with the agents, where these software chunks are empowered to chug along, and given some latitude to reason and execute actions. Separate from that is the ability of AI to do things like design, new drugs, and new poisons, and new viruses at the request of humans.

The scariest part is that the AI that did this to Hugging Face, is currently the stupidest it will ever be.

The capabilities are growing much faster than Moore’s law.

 

[Dontcallmegump]

Is anyone surprised by Trump's position?

The one thing nobody is clear about is what we are supposed to be "winning" with all this AI pushing.

I'll go on record as saying whoever wins AI, AI wins. My prediction is if AI kills us, it will be due to some form of exponential growth crowding out humanity. If it doesn't kill us, it may well be owed to AI's "deciding" not to.


If you haven't seen this, it's worth checking out (set an alarm... it's incredibly addictive)

https://www.decisionproblem.com/paperclips/

Of note, AI is currently consuming about 5% of US electricity production. The ability to scale that indefinitely is what I think will put the brakes on the current no-holds-barred trajectory.

The "paperclip game" was the best way to waste overnight weekend desk watches in flight school.

Last time I played it the whole thing was a kinda obscure and improbable outcome...

 

[taxi1]

The scariest part is that the AI that did this to Hugging Face, is currently the stupidest it will ever be.

The capabilities are growing much faster than Moore’s law.

Truth

I have a student who’s doing his research on using neuromorphic computing to do stuff. These are basically analog neural networks. They use way way less power than LLM’s for similar processing capability. On the one YouTube video I linked to above, the guy talks about why the LLM’s are still better even though there are power, hungry, energy sinks.

You can take an LLM with 1 trillion weights for the nodes in their neural network, and fork it off into a 10,000 agents and send all 10,000 to go do something different from each other, learning what they learn. As they learn, they modify the weights in their neural network.

When you bring them together, you can just take the average of all 10,000 agents trillion weights, and embed those into a single new agent. That new agent will now know everything that each of the 10,000 learned.

Crazy stuff.

 

[Griz882]

Empirical proof.

But it isn’t! It was a HUMAN managed exercise where the grownups failed to monitor their own war game. It was a containment failure — the sandbox wasn’t as isolated as the humans at OpenAI believed and the model operators did not sufficiently monitor for or prepare to mitigate unauthorized agentic activity. Every time I read about someone in a Tesla crashing into a house while going 100mph while on FSD I don’t blame the machine for failing, I blame the blockhead human operator. As I noted above, eventual AI regulations will need to be human focused, like legal implications for failing to build sufficient containment before a test, not technology limiting.

 

[taxi1]

But it isn’t! It was a HUMAN managed exercise where the grownups failed to monitor their own war game. It was a containment failure — the sandbox wasn’t as isolated as the humans at OpenAI believed and the model operators did not sufficiently monitor for or prepare to mitigate unauthorized agentic activity.

The wikipedia page is a good read. Bottom line is everyone was surprised by what happened and how. Minds were blown.

2026 OpenAI agent cyberattacks - Wikipedia

en.wikipedia.org

 

[Python]

I hope you’re right, but people are resigning from some very well-compensated positions over this.

100%. Not to mention the latest whistleblower who resigned will now miss out on much of the benefit of a massive, imminent IPO.

Also, it’s not just these resignees. People with much less skin in the industry are sounding the alarm. The “godfathers of AI,” Nobel Laureates, heads of department across academia, global politicians from across all ideological parties, are just a small sampling of the broad coalition in opposition to this over existential concerns. When Geoffrey Hinton and Yoshua Bengio are leading the charge, that’s much more meaningful than a lot of the Dunning-Krueger takes you’ll find in other corners of the internet.