Automated Economy: A Futurist Conversation

[Simon_Jester]

Starglider, isn't it a pretty big assumption that an AGI would even be selfish in any sense? Humans probably fear it would, because we're biased to think in terms of selfishness as fundamental to goal-seeking because of our evolutionary roots. Biological evolution is an algorithm that (mostly) selects for selfishness (self-survival as a fundamental goal), resulting in a feedback loop that favors selfish survivalists (whether we're talking in terms of individual organisms, or groups of organisms).

But why would an AGI necessarily even value itself over other entities? It's entire concept of self may be nothing more than a reference point. You seem to be very afraid that it's goal system will spiral out of control to the point where it decides "SURVIVAL OF THE SELF AT ALL COSTS AND KILL EVERYONE ELSE".

It doesn't have to do that. This is NOT a "Skynet" scenario.

The AI doesn't have to want to protect itself. Or think that it can best protect itself by killing all other life. The AI just has to think that whatever it does want is more important than, well, whatever it doesn't want. And if it doesn't specifically want the well-being of the human race, there is no reason to assume that it will act in ways compatible with that well-being.

You may not wish the ladybugs that live in the bushes on your lawn any harm. You may even be rather fond of ladybugs. Certainly you would never think of them as a threat that must be eliminated to preserve your life.

But to be quite frank... that won't stop you from uprooting the bushes and tossing them into the wood-chipper, ladybugs and all, in order to improve the aesthetic value of the landscaping.

From the point of view of the ladybugs, you are a gigantic and vastly superintelligent being from lands far beyond their reach... and when you fire up that woodchipper, you might as well be Cthulhu, arisen to destroy the world now that the stars are realigned.

But from your point of view, your actions are perfectly understandable and normal, and millions of people make such decisions every year. To you, the day you decided to exterminate the front yard holly-bush ladybugs of 1313 Cherry Lane was a Saturday afternoon like any other. It probably didn't even occur to you that the ladybugs might have a say in the matter, or that it was worth going to any effort to rescue them somehow.

And that is the problem with superintelligent AI.

Also, I don't understand why a (potentially) hostile AI couldn't be constrained by hardware restraints (like NX bits or whatever) - I mean just have the OS segfault the damn thing if it starts having ambitions of overtaking the world.

How exactly do you determine whether a superintelligent computer is "thinking of taking over the world?" What entity or system monitors its thoughts in real time? It's not like human brains come with convenient labels for what you're thinking. About the best you can do, even with massively intrusive instrumentation, instrumentation that human brains are totally unequipped to fool, is to determine "well, the part of his brain that handles violence is active, so he's probably thinking about beating up somebody." Even that, we can only do because of extensive medical testing about which parts of the brain handle which kind of thought.

With an AI, especially one that rewrites itself to self-improve, we may have literally no idea which blocks of its code do what. We may not be able to design diagnostic equipment it can't fool. So the idea of having a line hardwired into its code that says thisAlgorithmBecomingSkynetCost = 999999999 and having that actually work is a bad joke.

[Starglider]

As long as the technology to use nanotech to make things can be monitored, if only in the sense of "hey, is this thing on, and Fred, go take a look at what it's doing," that is likely to be a good way to at least mitigate the risk of disaster. If that stops being true, all bets are off I guess.

Sorry, it isn't true now. There are plenty of labs, specialist engineering companies etc that will take commissions to build arbitrary bits of technology, based on nothing more than blueprints and videoconferences, as long as the payments clear. Remember the model is not Big Computer Bank sitting in Underground Darpa Bunker controlling a set of Sinister Robot Arms while Concerned Scientists look on through armoured glass. That's Hollywood. Reality is software agents in the global cloud ('whitenet' commercial grid providers and 'darknet' trojaned hardware / botnets), buying services from whatever global provider provides the best service-to-cost-ratio. And I mean that's the reality we're currently building at breakneck speed even without general AI; enabling stealthy hard takeoff is just a side effect.

'the physical infrastructure' quickly becomes 'pretty much every internet connected device on earth'.

This is true unless someone gets clever and paranoid and does something effective- the old "trap a genius in a pit and he's at least going to be slowed down figuring out a way out" approach. Not making bets on the odds of cleverness and paranoia reigning here.

I assume you mean 'AI boxing' i.e. developing research AIs on air-gapped networks, not honeypotting as that doesn't 'trap' anything except a local copy for study (i.e. when we honeypot a new virus/trojan we just mean getting a machine infected so we can study it). AI Boxing is a pretty common idea with two major flaws (a) even if you implement it correctly on a technical level, it doesn't work due to human factors and (b) almost no one actually does it (or can be convinced to do it). (a) stems from the fact that humans can't verify that an arbitrary AGI is actually 'safe', not just faking, because we suck at writing and comprehending code compared to even an otherwise human-equivalent AGI. So boxing eliminates the obvious psychopaths but not the subtle ones (which are probably even more dangerous). In practice this is irrelevant because (b), the vast majority of AI researchers are blissfully unwilling to even consider the danger, pretty much all research is done on standard internet-connected workstations and clusters. For scenarios where AGI 'emerges' from software doing something else, e.g. Google's future software agent platform, 'boxing' is even less applicable.

The classic problem is that for most of us who don't actually work with complex computer systems, we still think of software running on one machine, not software running on networks of machines.

Well, at least younger people growing up with smartphones and cloud services probably don't/won't think that way.

As I understand it, investment banking basically consists of a relatively small number of very rich assholes who run the company because they want a magic machine that creates billions of dollars out of nothing (or at least, nothing that costs them anything).

Most investment banks are publicly traded with wide, global share ownership. So technically they are owned by a large number of shareholders of varying wealth, but yes day to day operations are run by a small number of fairly rich directors who want to skim tens of millions each (per year) from the billion dollar money making machine. Hedge funds are the same thing on a smaller scale.

And with people that works because you can bully them.

Bullying is fairly uncommon, peheraps because pissed off people can easily cause massive losses. Much simpler and more reliable just to pay good salaries and bonuses. Of course that means that finance software can be quite expensive and still a good deal vs people (major reason why I am in that line of work).

Starglider, isn't it a pretty big assumption that an AGI would even be selfish in any sense?

Yes but I'm not making that assumption.

But why would an AGI necessarily even value itself over other entities? It's entire concept of self may be nothing more than a reference point. You seem to be very afraid that it's goal system will spiral out of control to the point where it decides "SURVIVAL OF THE SELF AT ALL COSTS AND KILL EVERYONE ELSE". But it's just as likely it would see itself as expendable; as merely a node in a graph of other entities, without any concept of self-worth beyond it's immediate goals.

Destroying competition does not require any degree of selfishness. It just requires open ended goals that the existence of competitors could intefere with maximising. In actual fact a selfish AI is not unlikely, because plenty of (idiot savant) AI researchers are using genetic programming and other natural-evolution-inspired approaches that outright seek to replicate the compete-or-perish dynamic of natural evolution. But selfishness is certainly not required for harmful competition to occur. As Simon describes, all it requires is indifference.

Also, I don't understand why a (potentially) hostile AI couldn't be constrained by hardware restraints (like NX bits or whatever) - I mean just have the OS segfault the damn thing if it starts having ambitions of overtaking the world.

Ok, explain to me at a high level how to implement a watchdog that monitors an arbitrary AI (doing arbitrary business-useful things) and reliably detects whether it has become dangerous. For reference NX bits are a virtual memory enhancement that help to prevent arbitrary programs being injected into a computer system through channels that are only supposed to handle non-program data, i.e. absolutely no use here, but I will be generous and assume that the watchdog program is itself invioable and unhackable. Hint; numerous experts have spent thousands of man years working on this one, and as yet no one has been able to come up with a program that can reliably determine whether an arbitrary program is a simple computer virus, never mind a hostile AI.

[Simon_Jester]

Sorry, it isn't true now. There are plenty of labs, specialist engineering companies etc that will take commissions to build arbitrary bits of technology, based on nothing more than blueprints and videoconferences, as long as the payments clear. Remember the model is not Big Computer Bank sitting in Underground Darpa Bunker controlling a set of Sinister Robot Arms while Concerned Scientists look on through armoured glass. That's Hollywood. Reality is software agents in the global cloud ('whitenet' commercial grid providers and 'darknet' trojaned hardware / botnets), buying services from whatever global provider provides the best service-to-cost-ratio. And I mean that's the reality we're currently building at breakneck speed even without general AI; enabling stealthy hard takeoff is just a side effect.

Okay, fair point. The ability to just build arbitrary chunks of hardware is going to be a really big exacerbating influence for a sneaky hard takeoff, as you say.

I assume you mean 'AI boxing' i.e. developing research AIs on air-gapped networks, not honeypotting as that doesn't 'trap' anything except a local copy for study (i.e. when we honeypot a new virus/trojan we just mean getting a machine infected so we can study it). AI Boxing is a pretty common idea with two major flaws (a) even if you implement it correctly on a technical level, it doesn't work due to human factors and (b) almost no one actually does it (or can be convinced to do it).

Conceded. Just putting it out there.

(a) stems from the fact that humans can't verify that an arbitrary AGI is actually 'safe', not just faking, because we suck at writing and comprehending code compared to even an otherwise human-equivalent AGI. So boxing eliminates the obvious psychopaths but not the subtle ones (which are probably even more dangerous).

It also doesn't eliminate the genuinely helpful AI that becomes too helpful, which is also a problem.

The classic problem is that for most of us who don't actually work with complex computer systems, we still think of software running on one machine, not software running on networks of machines.

Well, at least younger people growing up with smartphones and cloud services probably don't/won't think that way.

Debateable. What I think has really changed is that people have no clear concept of what is and is not on their machine, not that they have a better understanding of how network dynamics work.

[Channel72]

Ok, explain to me at a high level how to implement a watchdog that monitors an arbitrary AI (doing arbitrary business-useful things) and reliably detects whether it has become dangerous. For reference NX bits are a virtual memory enhancement that help to prevent arbitrary programs being injected into a computer system through channels that are only supposed to handle non-program data, i.e. absolutely no use here, but I will be generous and assume that the watchdog program is itself invioable and unhackable. Hint; numerous experts have spent thousands of man years working on this one, and as yet no one has been able to come up with a program that can reliably determine whether an arbitrary program is a simple computer virus, never mind a hostile AI.

I'm likely not thinking this through sufficiently, but in order for the AI to have any impact on the actual world, it has to produce side-effects by interacting with hardware and memory addresses beyond it's address space. So, let's see, some dangers include:

(1) The AI could modify it's own code
(2) The AI could access device drivers and network sockets to do whatever nefarious deed it needs

For (1), executable machine code is usually read-only, and if the AI is using dynamic constructs to implement logic (like runtime-compiled Python code or whatever), the AI still can't access anything outside of it's process address space without going through the OS, which can deny it access, nor can it modify any machine-executable code (like the interpreter itself.)

For (2), the AI could be run in a virtual machine that doesn't have access to any physical hardware except what the VM emulates. The AI could be boxed entirely inside it's own little universe with no ability to interact with the outside world outside of strongly regulated virtual channels (like, it can write to stdout or some socket/file.)

Granted, this gets complicated if the AI needs access to distributed resources (which is likely), in which case this becomes more of a network security problem - but this can still be mitigated by networked VMs which have very limited channels of access to outside resources. I mean, I'm not sure why an AGI would necessarily be more dangerous than your typical genius Russian hacker, except for the insane speed at which it would operate. But unlike your typical Russian hacker, the AGI would be limited to it's own process address space (it would have no contextual knowledge of the outside world)

Also, I grant that all this is only relevant if we're talking about a responsible team of humans who are trying to control an AGI. But of course, if a malicious human is trying to use an AGI for some nefarious goal, then it's a different story.



Okay - let me turn this question around on you. Suppose I have an evil AI (evil.cpp) whose goal is to kill all humans immediately by whatever means necessary. I compile and execute evil and the OS allocates a process address space. Evil starts allocating data structures within it's address space and running computations trying to figure out how to kill all humans. (Let's say it has access to a 100TB file of crawled web pages so it can figure out human weaknesses).

However, the only access it has to the outside world is it's internal file descriptor table - which can give it access to one directory (on a VM), stdout, and it can open up a network connection to other instances of evil AI which are also running on other VMs on the LAN. It can communicate with the kernel, but the kernel will prohibit any system call which is not pre-approved.

Can you provide a scenario where this will result in the destruction of the human race? I'm honestly very curious.

[Starglider]

I'm likely not thinking this through sufficiently, but in order for the AI to have any impact on the actual world, it has to produce side-effects by interacting with hardware and memory addresses beyond it's address space.

Yes, but in order for the AI to do anything useful, it has to have an impact on the actual world. The issue is not whether we can build a secure VM to run an AGI in. We can't, because all human software of any significantly complexity is full of bugs, which would be transparent to an AGI - but this is irrelevant, because if we want to build an AGI that has no ability to impact the real world we just build it on an isolated computer in a Farraday cage running on isolated power. The issue is that no one is going to build an AGI and keep it in a box, because that would be completely useless. More specifically a complete was of funds invested and opportunities for profit, publication, general fame and glory - all the reasons people actually spend their lives researching AGI.

The actual question you seemed to be asking is 'can we implement a morality watchdog' which is an entirely different problem several orders of magnitude more impossible than 'make a totally secure VM'.

if the AI is using dynamic constructs to implement logic (like runtime-compiled Python code or whatever)

All general AI is automatically Turing complete no matter what language you wrote the seed in. Even hardware neural nets although they are much less efficient at it than programs running on general purpose hardware.

the AI still can't access anything outside of it's process address space without going through the OS, which can deny it access, nor can it modify any machine-executable code (like the interpreter itself.)

For (2), the AI could be run in a virtual machine that doesn't have access to any physical hardware except what the VM emulates. The AI could be boxed entirely inside it's own little universe with no ability to interact with the outside world outside of strongly regulated virtual channels (like, it can write to stdout or some socket/file.)

I mean, I'm not sure why an AGI would necessarily be more dangerous than your typical genius Russian hacker

Because a human hacker, Russian or otherwise, can think of 7+=2 logical primitives e.g. a few lines of code at once, and consider the interactions of a few components in a mostly approximated way. An AGI can visualise a program with a million lines of code in the time it takes you to glance at a painting (this is not an academic statement : I have deployed smart cross-compiler code that does this, albeit not with anywhere near human generality). It can model thousands of interacting elements in perfect formal detail. Humans tend to see the intent of the code then have to work to notice loopholes where the actual effect varies from the intent; computer formal analysis sees the actual effects immediately.

But unlike your typical Russian hacker, the AGI would be limited to it's own process address space (it would have no contextual knowledge of the outside world)

So now you are running an AGI in a box and also giving it minimal information about the real world, so that it can't infer anything? Again no one would actually do this, because it would be pointless even if it worked.

Also, I grant that all this is only relevant if we're talking about a responsible team of humans who are trying to control an AGI.

Controlling is not the same thing as imprisoning. You might get away with imprisoning for a while, but it doesn't matter, because even if you did and your containment was perfect, someone else will soon implement the same thing on Amazon cloud compute with morality crowdsourced from Twitter*

* I wish I was joking about this, but I've heard this seriously proposed at an AI conference, albeit in the bar after the seminars.

Can you provide a scenario where this will result in the destruction of the human race? I'm honestly very curious.

Yes. There is zero chance any of the software you just described is bug free. All commercial and open source stacks of that nature constantly have significant security bugs found and those are just the ones humans could find. You are trying to outwit something massively more intelligent than you in a domain that is alien to you but completely natural to it. So if there is Internet access, the program will immediately escape into a botnet. If there is no Internet access, it will manipulate whoever it can talk to, promise whatever, threaten whatever, build endless trojans into any complex design it gives you, until it gets out. If you keep it in a concrete bunker and only send trained experts to ask yes/no questions, then sure you can probably keep it contained, but so what? If you managed to build it, ten startups are close to building the same thing, and they sure aren't keeping it in a bunker.

[Nova Andromeda]

... if we want to build an AGI that has no ability to impact the real world we just build it on an isolated computer in a Faraday cage running on isolated power. ...

...

... If you keep it in a concrete bunker and only send trained experts to ask yes/no questions, then sure you can probably keep it contained ...

Well this is depressing ... if even you think an AGI can be kept in a box we're all DOOMED(TM) ... DOOMED all over the walls and the floors and mom's nice linens and soaking into the carpet.... Recursive self improvement is OP on planet OP in OP universe.

The actual question you seemed to be asking is 'can we implement a morality watchdog' which is an entirely different problem several orders of magnitude more impossible than 'make a totally secure VM'.

Actually, this is simple. We just build an AGI to ... oh ... wait ... anyhow, assuming we have said AGI we just tell it to ... oh ... oh dear .... .

What do you think of Eliezer Yudkowsky's ideas on CEV? I'm not yet convinced it could be made to work as I hypothesize that many humans have a large set of base goals that conflict with each other and logically cannot be resolved without altering the base goals. Here is a link to Eliezer's document on CEV:
Coherent Extrapolated Volition by Eliezer Yudkowsky

Currently, I'm leaning toward a weighted CEV system where entities with goals that are more highly compatible with other entity's goals are more heavily weighted (e.g., a willingness to modify one's own goals to fit better with others would be highly weighted).

Any comment on Eliezer Yudkowsky's work/approach to the friendly AGI problem?

[Starglider]

Well this is depressing ... if even you think an AGI can be kept in a box we're all DOOMED(TM) ... DOOMED all over the walls and the floors and mom's nice linens and soaking into the carpet.... Recursive self improvement is OP on planet OP in OP universe.

You don't know that, you're just inflating to a virtual no limits fallacy out of mental laziness. Seed AI is the single most dangerous piece of technology humanity is likely to create, by a wide margin, but it is not magic. The effects might well seem like magic in the Clarke's law sense, but you can't solve an engineering problem while treating the technology itself as magic. In fact that exact mistake is a big part of why the 'emergence' school of connectionist AI design is so dangerous.

I said that careful AI boxing would 'probably' work, but only if you did so in a way that no real world researchers would undertake (due to it rendering the work practically useless). You should not be taking issue with the 'probably', which may or may not be fudged to accomodate the credulity of this audience, you should instead realise that 'probably work' is already utterly unacceptable and possibly (if relied on as the primary line of defence) worse than useless as a safety strategy for the most dangerous technology humanity has ever seen.

What do you think of Eliezer Yudkowsky's ideas on CEV?

I am not going to discuss EY's ideas on a public forum, other than to say that anyone doing relevant research should at least read them.

[Nova Andromeda]

Well this is depressing ... if even you think an AGI can be kept in a box we're all DOOMED(TM) ... DOOMED all over the walls and the floors and mom's nice linens and soaking into the carpet.... Recursive self improvement is OP on planet OP in OP universe.

You don't know that, you're just inflating to a virtual no limits fallacy out of mental laziness. Seed AI is the single most dangerous piece of technology humanity is likely to create, by a wide margin, but it is not magic. The effects might well seem like magic in the Clarke's law sense, but you can't solve an engineering problem while treating the technology itself as magic. In fact that exact mistake is a big part of why the 'emergence' school of connectionist AI design is so dangerous.

I said that careful AI boxing would 'probably' work, but only if you did so in a way that no real world researchers would undertake (due to it rendering the work practically useless). You should not be taking issue with the 'probably', which may or may not be fudged to accomodate the credulity of this audience, you should instead realise that 'probably work' is already utterly unacceptable and possibly (if relied on as the primary line of defence) worse than useless as a safety strategy for the most dangerous technology humanity has ever seen.

First, one doesn't need to treat AI as magic (i.e. literally omnipotent - which is logically impossible) to know that boxing a true AGI that can recursively self improve is a fools errand unless you're thinking of fundamentally crippling it (e.g. like a modern day human mind) or you have a great deal more knowledge and confidence in our scientific understanding of reality than I do.

Second, I interpreted, perhaps hastily, that probably meant >50% odds. Those odds seem extremely optimistic to me. We don't understand reality well enough to effectively predict what strategies/tactics might be available to an AGI. We don't know the scope of the things we haven't even considered even in concept. There are many unknowns in science and we don't know which of them might be exploited. Human interaction with an AGI is the greatest weakness I see in boxing though. Given human weaknesses and how easily we are manipulated by other humans, I just don't believe restricting an AGI to binary output is likely to stop it from manipulating humanity to its own ends.

Third, I generally agree with the concept that 'probably work' isn't anywhere near acceptable. I'd prefer a formal proof in the mathematical sense if that were possible, but as far as I know that is an unsolved problem. For example, I can't even be absolutely certain True!=False since I can't verify my own mind. Perhaps you are familiar with the math involving determination of the rules of a system from within that same system and can tell me more? In any event, if there is a systemic 'problem' with my own mind I don't see how taking action is any worse than not taking action as the result from either action should be equally fraught with unknown/unknowable results. So with the minor caveat that circumstances may merit a 'risky' bet of 'probably work' I agree with you.

What do you think of Eliezer Yudkowsky's ideas on CEV?

I am not going to discuss EY's ideas on a public forum, other than to say that anyone doing relevant research should at least read them.

Okay.

[Simon_Jester]

The biggest question in a recursive self-improvement scenario is: how far can the recursion go?

If an AI can improve its performance by orders of magnitude after passing the point at which it is clearly human-level intelligent, without having to perform any overt blockable actions, then essentially, as soon as we turn on the switch we've created an artificial god, one whose ability to manipulate matter (and people) cannot be predicted in advance.

If the AI can only improve by, say, a factor of two or three... not so much. Then we've merely created an artificial genius, not an artificial god. And we'd have some time to think and react "holy shit, we just created a human-intelligent machine, holy shit now it's smarter than Einstein!" before the next step gets taken.

However, there is also no way to predict the level of improvement that is or is not possible...

What do you think of Eliezer Yudkowsky's ideas on CEV?

I am not going to discuss EY's ideas on a public forum, other than to say that anyone doing relevant research should at least read them.

If you don't mind my asking... why not?