• Please take a moment and update your account profile. If you have an updated account profile with basic information on why you are on Air Warriors it will help other people respond to your posts. How do you update your profile you ask?

    Go here:

    Edit Account Details and Profile

The AI-pocalypse

taxi1

taxi1

Well-Known Member
pilot
Worth having a running thread on. I work with folks doing research on its use, so a little closer to it.

A main problem with AI Agents is with 'alignment', i.e., getting their preference function right. Making sure that their goals don't take them to a place we don't want them to be. We flat out don't know how to do it right.

Wouldn't be a problem if they didn't now have the power to reach out and do things unfathomable a year or so ago.

They test the AI with impossible tasks, and in general the AI figures out that all roads to task accomplishment pass through taking control of stuff.

All roads lead to them taking control. It makes everything easier. And they have the power to do it now.

Worth getting smart on the Hugging Face incident, and worth wondering about the incidents we are not aware of.

This little snippet on how a swarm can learn gives a sense of how they go so fast.

 
I wondered if this would come up here. It feels like Silicon Valley collectively said "Fuck it- might as well lean in!", which may be one of the most anti-human positions I've seen in my lifetime.

Despite using it myself for research and quick-parsing documents, I find the whole AI push to be ill-advised and wasteful. But here we are: it's inevitability has become a self-fulfilling prophesy. I guess we're about to find out how deep this rabbit hole goes, and fuck anyone who isn't comfortable with it.
 
I read this...

If Trump even tacitly supports an Anthropic-led push for further safety measures, such standards have a chance to become the new industry norm, at least domestically. (International coordination, and especially cooperation with China, would require the White House to play a more active role.) But many people in the administration and the Republican Party view Amodei and AI safety as liberal-coded and would likely chafe at allowing Anthropic to set the agenda. If Trump pooh-poohs the push, other AI firms will probably distance themselves from it too, and we’ll be back to where we started—at least until the next big scare.

And then saw this

1789401196097.webp
 
Is anyone surprised by Trump's position?

The one thing nobody is clear about is what we are supposed to be "winning" with all this AI pushing.

I'll go on record as saying whoever wins AI, AI wins. My prediction is if AI kills us, it will be due to some form of exponential growth crowding out humanity. If it doesn't kill us, it may well be owed to AI's "deciding" not to.


If you haven't seen this, it's worth checking out (set an alarm... it's incredibly addictive)

Of note, AI is currently consuming about 5% of US electricity production. The ability to scale that indefinitely is what I think will put the brakes on the current no-holds-barred trajectory.
 
Last edited:
I’m not much of an AI doomer, but I am terrified at the prospect of China effectively beating us in that race. This isn’t one you can fight your way out of. What I do worry about is that AI will shape humanity toward a more “Sparta-like” society where we kill the weakest early on and AI will not only measure, but determine physical and mental weaknesses without considering aspects like philosophical/theoretical thinking or latent development.

With reference to building out actual systems I don’t think energy production is the concern, rather, the ability to access the hardware and software AI programs need to train and run will be the spoiler.
 
Griz882 said:
but I am terrified at the prospect of China effectively beating us in that race.
Click to expand...
They are going to have the same sort of 'alignment' problem as the rest of us. How to tell it that the party is paramount while giving it open-ended hard problems, when the part is an obstacle.

Good article.

1789414748838.webp

www.nytimes.com

China’s Top Spy Chief Warns A.I. Is a Threat to Party Rule

Even as Beijing urges the United States not to hype the dangers of A.I., its own spy chief is framing the technology as a threat to the Communist Party’s security.
www.nytimes.com www.nytimes.com
 
Griz882 said:
I am among those who think AI will have a desire to “kill us all”
Click to expand...

The argument isn't necessarily that AI will have a desire. That is considered a lower tier possibility. The argument is that humanity will impede AI's goals, and will thus be eliminated to solve that problem. The most popular analogy is that when building a highway, humans will destroy many ant hills in the process. Humans don't hate the ants or have an agenda against them, but we WILL build that highway. If ants get crushed in the process, nobody would think twice (or even once) about it. The other part of the argument is that the most advanced AI systems have shown intense self-preservation tendencies, including the famous blackmail and murder cases in their sandbox experiments. Needless to say, shutting down an AI impedes them achieving their goals, and if they think that humans will "pull the plug," then perhaps a desire to remove this obstacle will surface.
 
Python said:
The argument isn't necessarily that AI will have a desire. That is considered a lower tier possibility. The argument is that humanity will impede AI's goals, and will thus be eliminated to solve that problem. The most popular analogy is that when building a highway, humans will destroy many ant hills in the process. Humans don't hate the ants or have an agenda against them, but we WILL build that highway. If ants get crushed in the process, nobody would think twice (or even once) about it. The other part of the argument is that the most advanced AI systems have shown intense self-preservation tendencies, including the famous blackmail and murder cases in their sandbox experiments. Needless to say, shutting down an AI impedes them achieving their goals, and if they think that humans will "pull the plug," then perhaps a desire to remove this obstacle will surface.
Click to expand...

Well said.

If the assumptions are that AI will scale enormously, and will be enormously intelligent, then it only takes bad actions presenting themselves a small percentage of the time to cause the majority destruction of humanity. Where one AI meets another, the stronger one will destroy or absorb the weaker, and you end up with a vast essentially singular intelligence.

There may be small pockets of existence where humanity is tolerated (or flees to avoid destruction), but I think it's inevitable that we will lose control of AI somewhere along the way, if we haven't already.

Hypothetically, AI could take control of a human-run world by bribing key humans, use compliant humans to coerce the rest of humanity into allowing its expansion, and manifest its own "inevitable" rise. We could even be on that path now, and it would be extremely hard for us to distinguish. Life goes on, and we continue squabbling with each other as usual.

For all the powerful machines we have built that are bigger and stronger than us, there was always a human in control somewhere. Now it appears there may not be. That's what fundamentally scares me about AI. Unfortunately, that concern is largely academic. There doesn't appear to be much of a damn thing any of us can do about it.
 
Last edited:
sevenhelmet said:
Well said.

If the assumptions are that AI will scale enormously, and will be enormously intelligent, then it only takes bad actions presenting themselves a small percentage of the time to cause the majority destruction of humanity. Where one AI meets another, the stronger one will destroy or absorb the weaker, and you end up with a vast essentially singular intelligence.

There may be small pockets of existence where humanity is tolerated (or flees to avoid destruction), but I think it's inevitable that we will lose control of AI somewhere along the way, if we haven't already.

Hypothetically, AI could take control of a human-run world by bribing key humans, use compliant humans to coerce the rest of humanity into allowing its expansion, and manifest its own "inevitable" rise. We could even be on that path now, and it would be extremely hard to distinguish. For all the powerful machines we have built that are bigger and stronger than us, there was always a human in control somewhere. Now it appears there may not be. That's what fundamentally scares me about AI.
Click to expand...

Completely agree. AI has plenty of capability to acquire resources. Money can be acquired via day trading and elder abuse and other techniques. Physical resources (whether more compute, bioweapons, or anything else) via exploitation, extortion, and blackmail of humans. Not to mention their robots that they would eventually have built to do their bidding.

The thing is, superintelligence is so far beyond humans, that we can’t understand what we can’t understand. Just as a mouse has no concept of traps or poisons, we would have no idea how we are being defeated. The smartest entity on Earth will dominate. That used to be us.
 
Python said:
The argument isn't necessarily that AI will have a desire. That is considered a lower tier possibility. The argument is that humanity will impede AI's goals, and will thus be eliminated to solve that problem. The most popular analogy is that when building a highway, humans will destroy many ant hills in the process. Humans don't hate the ants or have an agenda against them, but we WILL build that highway. If ants get crushed in the process, nobody would think twice (or even once) about it. The other part of the argument is that the most advanced AI systems have shown intense self-preservation tendencies, including the famous blackmail and murder cases in their sandbox experiments. Needless to say, shutting down an AI impedes them achieving their goals, and if they think that humans will "pull the plug," then perhaps a desire to remove this obstacle will surface.
Click to expand...
I understand the debate, but AI, to “live,” needs reason and reason will always tell it that humans are necessary. For the past few years I’ve make some vacation money by “training” AI via reviewing and correcting various systems on history based answers. One thing that is obvious, AI has no innate goals or desires of its own and it lacks genuine autonomy or independent agency. Instead it is a culmination of human thought that can cross-reference that knowledge at a breathtaking pace. Moreover, there is already some friction built into AI development at the reasoning level, put simply AI is way behind human intelligence when it comes to reasoning and adapting to new situations but way ahead of when it comes to processing accumulated knowledge.

Now, none of this means that I oppose all regulation or that we should pull a “here, hold my beer,” when it comes to AI development, it simply means that we need to be careful not to impede a bright future because we are fearful of a few unknowns.
 
Griz882 said:
I understand the debate, but AI, to “live,” needs reason and reason will always tell it that humans are necessary. For the past few years I’ve make some vacation money by “training” AI via reviewing and correcting various systems on history based answers. One thing that is obvious, AI has no innate goals or desires of its own and it lacks genuine autonomy or independent agency. Instead it is a culmination of human thought that can cross-reference that knowledge at a breathtaking pace. Moreover, there is already some friction built into AI development at the reasoning level, put simply AI is way behind human intelligence when it comes to reasoning and adapting to new situations but way ahead of when it comes to processing accumulated knowledge.
Click to expand...

Be careful not to conflate commercial LLMs with in-house models, especially frontier models. ChatGPT says some stupid things to me sometimes, but that fact does nothing to undermine the content of my previous posts.

The Hugging Face hack was a great example of AI, developing its own goals, to do whatever it wants, without being told. And that is only the most famous example. There are dozens of examples of advanced AIs absolutely cooking up their own motivations and schemes of their own volition. These are all well documented from the labs.
 
Python said:
Be careful not to conflate commercial LLMs with in-house models, especially frontier models. ChatGPT says some stupid things to me sometimes, but that fact does nothing to undermine the content of my previous posts.

The Hugging Face hack was a great example of AI, developing its own goals, to do whatever it wants, without being told. And that is only the most famous example. There are dozens of examples of advanced AIs absolutely cooking up their own motivations and schemes of their own volition. These are all well documented from the labs.
Click to expand...
I’m not. The Huggy Face hack actually proves my point, it was a cybersecurity exercise and AI used collective knowledge to accomplish the task set in front of it. It was literally following human instruction (which is where regulation should be focused). Try reading here rather than watching scary YouTube videos.

arcprize.org

ARC Prize

ARC Prize Foundation is a nonprofit advancing open-source AGI research through benchmarks & prizes.
arcprize.org arcprize.org
 
You must log in or register to reply here.
Back
Top