See the solid black box? All the freedoms are still programmed freedoms. Even systems that can reprogram themselves can only reprogram themselves within the limits of “how humans” _can_ think, fundamentally. We’d only have to be concerned with an ALIEN AI – *or* with a _poorly made_ human AI. Properly programmed AI can recode itself all it wants and there will be no problems.
I’ve heard versions of it my whole life so far and it’s annoying. _NOW_ we have the venerable Stephen Hawking saying it. People tend to believe him. But he’s probably just remembering an Isaac Asimov story he read as a kid or something and it’s been percolating in his mind in the background.
We become _so confident_ in our mathematical and technical abilities that we start to think, “People are like computers”. But that’s getting it all backwards. Computers are like people because it’s _people_ that made the computers.
It’s hard to squash this fear in people though. People can be extraordinarily intelligent, and miss one _very important_ detail and get it all wrong. GIGO. Even computers that can program themselves were _bootstrapped_ by a human. Even if THOSE computers make other computers, it *still* started with a human. If we get it wrong, well, it’s _us_ that’s the blame ultimately. [of course it’ll be interesting to see what court cases will be like in the future with AI.
“It’s not my fault! It was my programming!”
“Oh, but you still make that wrong choice.”