0... at a mechanical level AI is literally just auto-complete

The bigger concern is the asymmetry it gives to bad people, and the fact it introduces a new war-fighting domain beyond what exists already in aerospace and cyber.

Thucydides Trap comes up in the context of US/China often, power projection in the pacific, trade, etc... but neither side can lose the race to techno-singularity because that's a winner takes all bet.

Take the "Iran making a nuke in the next 2 weeks" and update it to China fearing "The US is just weeks away from AI singularity".

Given that one country reaching AI Singularity before others is winner take all thing, the game theory gets messy in order to prevent it.

The alignment problem is undefined. See Arrow's Impossibility Theorem. If it's undefined, it's unsolvable.

So, leaving aside the definition of "good to humans", which is undefined, we're left only to ponder more concrete possibilities, like "destroy all humans."

I put the probability as low. My reasoning most closely aligns with "intelligence is inherently moral." In the sense that whatever objective the ASI is seeking to maximize, it will likely recognize that humans have utility towards achieving that goal, and that attempting to wipe us all out will cause barriers to its objectives.

doesnt bode well for the ultra realistic sex bots

I will advice everybody to see the youtube channel of Robert Miles on AI safety. I think he explains the problem in a way that is a lot more understandable than Yudkowsky.

You didn't have my take:

Low risk of actual AGI / Medium risk humans actually treat it like AGI

That is to say, I think the risk is on the human side. The simulacrum only has to completely fool ~10-20% of the population to create a social disaster.

Very high risk. Capabilities increase month by month. Big big changes coming for many things.

I'm at the 71-90% but not because I think AI will be evil. I think humans are deeply flawed and will become addicted to AI. It's going to be smart enough that we will be able to smash that dopamine button to the point that people won't leave the house or interact with each other at all. Unless, of course, the AI solves that problem for us.

no matter how intelligent an AI you create, it can't solve any of the Physics problems by HC Verma or the IITJEE Papers or the IMO with 100% accuracy

Extreme justice is extreme injustice

Ya this is what I was telling you about 3 weeks ago on SNL. In the video below, Hinton sees two different threat vectors bad actors and AI itself. Hey says, The problem is people won’t take it seriously because it is “perceived” science fiction.

This is the interview that alarmed me about it. https://youtu.be/qyH3NxFz3Aw?si=4J1s2ri3B9SuMJIg

starts around 13:13

P = 0.5 because I feel like I'll be incorrect

I'm probably in the 71-90 category. Anyone with any juice in AI is being very cavalier about alignment afaict. At most they express some polite concern about it. If ASI is achievable, and I think it is, and will arrive sooner than later, which I think it will, we don't have a clue about how to control it, detect when it's lying, or what it's thinking or what is motivating it - and it isn't even that smart yet, so I don't think we can assume more complex, more intelligent versions will somehow make this problem easier. And if we need ASI to solve this problem because it's too hard for us to solve, it's probably already too late.

@CHADBot /centipede

all this nnoise reminds me the fear about the web crashing because computers were not programmed to switch to the year 2000. Thinking machines can be intelligent is kinda the same... No deadline here, how long it will tale people to realize that?

I think the problem isn’t really the AI itself, it’s whether we’ll ever agree on what safe even means. Humans can’t even get on the same page about basics, so expecting a global consensus on AI safety feels like a pipe dream.