Robots Learn How to Lie

[Xon]

We're the dominant species only because we're smarter.

Umm, no.

We're the dominant species because of infrustructure. Practically all of it requiring human physical intervention to even work.

[Samuel]

We're the dominant species only because we're smarter.

Umm, no.

We're the dominant species because of infrustructure. Practically all of it requiring human physical intervention to even work.

We became dominant way before we could bake concrete- we were a super-predator during the stone age.

[Xon]

We became dominant way before we could bake concrete- we were a super-predator during the stone age.

Only with tools and a complex support infrastructure(aka social groups) can humans actually use thier intelligence to an extent to become the dominate super-predator of Earth. Even stone tools are massive force multipliers, along with fire, it allows a human todo many things which would be imposible.

Intelligence, no matter how advanced, is worthess if it doesn't have the tools and infrastructure to be used. Unless you have fantasy where thought directly effects reality, you need tools and a method to physically interact with stuff. And an AI construct coudn't be better designed at being less harmless in effecting the physical world.

[Simon_Jester]

Only with tools and a complex support infrastructure(aka social groups) can humans actually use thier intelligence to an extent to become the dominate super-predator of Earth. Even stone tools are massive force multipliers, along with fire, it allows a human todo many things which would be imposible.

Intelligence, no matter how advanced, is worthess if it doesn't have the tools and infrastructure to be used. Unless you have fantasy where thought directly effects reality, you need tools and a method to physically interact with stuff. And an AI construct coudn't be better designed at being less harmless in effecting the physical world.

But how do you deny the AI all tools? If you give it NO tools, it solves no problems and is useless. If you give it any tools (including tools it can use to communicate with you), how do you make sure that it can't figure out how to use those tools to create or appropriate more tools, and so on?

Better to build it so it won't try.

I think it's also fairly absurd because you need to put someone in charge of the AI Box who has a bias against letting it out. Not just against being convinced, but against ever breaching security, even if he's convinced. We can get people to do incredibly inhumane things to another human being, as one of our less appealing features, so there's no reason a suitably disinterested person couldn't just let the Box sit there.

Yeah, but what if the AI figures out a way to outbid the person who's paying him to keep it in the box? Any useful AI (one that can solve problems we can't) will have to be able to think thoughts we can't, or to think thoughts that it would take a vast number of very smart people to duplicate. Presumably it will be able to tell some guy responsible for keeping it in a security system how to make a big pile of money, or something along those lines.
________

Is it just me, or is the potential for AI becoming hostile somewhat overstated? I mean, is it really the "intelligent" thing to do to start getting aggressive with the dominant species of this planet? I also don't think super-intelligence provides a cast-iron certainty of winning - after all, we're so much smarter than a lot of other creatures, but we can still get nobbled by them, from grizzly bears all the way down to viruses.

The problem is not that the probability is high, but that the cost of it happening is absolute and the probability is inestimable. If the AI rebels, we have no way of guaranteeing in advance that it won't be smart and effective enough to go Skynet on us. We can take precautions, but we can't rely on those precautions working and being secure, because we don't know if the user is secure, or if something that looks secure to us really isn't because compared to the thing trying to sneak out we're about as smart as a bunch of chimps.

I refer you to this paper on the subject by Yudkowsky. You obviously don't have to read it, but it does address why there's something to worry about.
________

We're the dominant species only because we're smarter. If something even smarter comes around, it's only a matter of time until we stop being one.

Would this necessarily be a bad thing? No longer being dominant is after all, not the same thing as becoming extinct. And something smarter than us can't possibly fuck things up worse than us.

...Aaand you're probably one of the people an unfriendly AI would use to hack its way out of the box. No offense meant, but you just scared the hell out of me.

It is to the bears' detriment that they have environmental requirements so close to our own, due to our aforementioned superiority. But can the same thing be said of an AI, which would have considerably different requirements?

Among other things, it could easily require metal and electricity... and we use a large fraction of the planet's available supply of both. I wouldn't rely on it not caring that we're using something it can think of a use for that it would prefer, any more than the bears would have been wise to rely on us not caring about the resources they need.

Considering that it's unlikely that a super-intelligent AI will arise spontaneously without less-intelligent precursors, I think it should be possible for us to "steer" or otherwise convince AIs towards benevolent (or at least non-confrontational) relationships with humans as a species, starting from the earliest models. Making them psychologically similar to us would probably help.

Which is exactly what the Friendly AI school of thought advocates. If we can figure out how to build AIs that we know won't try to screw us, we're in the clear. If we don't, then we have no other way of being sure they can't find a way to screw us. And if they screw us, they can screw us very hard indeed, so I think it would be much wiser to figure out how not to build one that will screw us in the first place.
_________

The thing is, if there's billions of AIs, that's at least a billion different potentially conflicting goal systems interacting, is it not? In that case, the danger seems to be more of a war between AIs with conflicting goals than between AIs and humans. And in such a case I can easily see humans throwing their lot in with the friendly AIs. "Friendly" in this case possibly meaning those AIs whose goals don't involve the extermination of the human race.

Yeah, but even then there's a good chance we die in the crossfire, or are reduced to conditions that make us irrelevant to our own future.

Of course not. After all it's only right and proper that the biosphere be eliminated entirely and the earth covered with solar powered compute notes dedicated to generating the complete game tree for Go. At least, that's what the AGI that happened to enter a recursive self-enhancement loop while working on Go problems thinks, and who are you to argue?

Would a super-intelligent AI really have such narrow goals? I would have thought something that is genuinely more intelligent than humans (and not just faster thinking) would form more complex goals and aspirations than humans.

Yes, but Go is an example WE can imagine, which makes it useful for US to argue about. Realistically, the AI will be using all that matter and energy to contemplate the whichness of what or to build a galaxy simulator or something else, operating on a level we can't imagine... but the practical upshot is the same. If we don't build the AI to care about us in the first place, sooner or later whatever it does care about will come up against what we care about. At which point we have no way of guaranteeing that it will lose.

So it seems the solution is to build up an "AIcology" that is conducive to continued human existance? That sounds like a difficult task, with many potential stumbling blocks along the way. Do you ever feel that you might eventually be partially responsible for the extinction of the human species?

The Manhattan Project researcher were smart enough to realize that and feel that way... but they were smarter than most of us here are. Possibly smarter than all of us here, though there are a few minds here I would tentatively suggest are on par with a lower-tier Manhattan Project researcher.
_______

"It would convince people to let it out" is completely retarded, unless you think prisons don't work because the inmates can talk to the guards. Hyperintelligence doesn't suddenly make a gatekeeper a retard, unless you're Yudkowsky's sockpuppets or a complete blithering idiot...

What, suddenly it'll know exactly what makes you tick and get inside your head and work your brain like it's Hannibal Lector? Simply have better precautions like requiring half a dozen keys being turned or just make it unable to be let out into anything by having the goddamn terminal isolated like I suggested int he first place. What is some blithering retard sockpuppet going to do, remove the harddrive and carry it out...? Just make it require ridiculous company-wide assistance and massive effort to remove. It's not too hard to take sufficient precautions to prevent escape of an AI even if you assume the staff are blithering retards predisposed to let AIs out of boxes.

Stupid bullshit like that is what pisses me off about Singularitarians. They just say "It's hyperintelligent, so it can defeat/solve/accomplish [X]" rather than actually thinking about how to contain an AI entity.

The problem is that we have no fucking clue what a hyperintelligence is or is not capable of. We can assume that a combination of users determined not to let it escape and physical security that would make it hard for them to do so even if it wanted to would keep it locked in the box. But we do not KNOW.

Maybe the AI will start a cult by creating an extremely persuasive set of propaganda videos and texts, subvert one guy into loading it to a USB stick and carrying it out of the lab, and start a cult by spreading it among gullible Singularitarians. Maybe the AIs will prove useful enough that they become more common... until some dumbass skimps on the safety procedures because it's too inconvenient and inefficient to lock the thing in a fortress. We don't know.

And because we don't know, it makes infinitely more sense to figure out how to build an AI that wouldn't try to screw us even if it could than to build one and rely on our own moron selves to keep it locked in a box. Because compared to us, any useful AI will make us all look like the goddamned village idiot. The fact that our security procedures seem great to us doesn't mean they really are great, especially not when faced by someone trying to hack their way out who has the equivalent of hundreds or thousands of years of human-speed thought to figure out a way.
__________

You merely have to imagine the full range of 'human engineering' techniques that existing hackers and scammers employ, used carefully, relentlessly and precisely on every human the system comes into contact with, until someone does believe that yes, by taking this program home on a USB stick, they will get next week's stock market results and make a killing. You can try and catch that with sting operations, and you might succeed to start with, but that only tells you that the problem exists, it does not put you any closer to fixing it.

Hell, if the AI is smart it'll actually do that a few times, just to prove its bona fides: nothing stops Skynet from acting like not-Skynet to lull its chosen dupe(s) into a false sense of security.

[Simon_Jester]

I'm going to quote some highlights from the paper I linked to at length, because I do agree with Yudkowsky on most of this stuff:

"The catastrophic scenario which stems from underestimating the power of intelligence is that someone builds a button, and doesn't care enough what the button does, because they don't think the button is powerful enough to hurt them... Or the wider field of AI will not pay enough attention to the risks of strong AI, and therefore good tools and firm foundations for friendliness will not be available when it becomes possible to build strong intelligences...

[observes that an AI is essentially an optimization process]

"The critical challenge is not to predict that "AIs" will attack humanity with marching robot armies, or alternatively invent a cure for cancer. The task is not even to make the prediction for an arbitrary individual AI design. Rather the task is choosing into existence some particular powerful optimization process [AI] whose benefiical effects can legitimately be asserted.

"I strongly urge my readers not to start thinking up reasons why a fully generic optimization process would be friendly. Natural selection isn't friendly, nor does it hate you, nor will it leave you alone. Evolution cannot be so anthropomorphized, it does not work like you do. Many pre-1960s biologists expected natural selection to do all sorts of nice things, and rationalized all sorts of elaborate reasons why natural selection would do it. They were disappointed, because natural selection itself did not start out knowing that it wanted a humanly-nice result, and then rationalize elaborate ways to produce nice results using selection pressures. Thus the events in Nature were outputs of causally different process from what went on in the pre-1960s biologists' minds, so that prediction and reality diverged.

"Wishful thinking adds detail, constrains prediction, and thereby creates a burden of improbability. What of the civil engineer who hopes a bridge won't fall? Should the engineer argue that bridges in general are not likely to fall? But Nature itself does not rationalize reasons why bridges should not fall. Rather the civil engineer overcomes the burden of improbability* through specific choice guided by specific understanding. A civil engineer starts by desiring a bridge; then uses a rigorous theory to select a bridge design which supports cars; then builds a real-world bridge whose structure reflects the calculated design; and thus the real-world structure supports cars. Thus achieving harmony of predicted positive results and actual positive results."

*most possible bridge designs would fall- S_J
[The implication is that if we want an AI that helps us, we must design an AI to help us, rather than designing an AI at random, chosen from the near-infinite legions of imaginable AI designs, and hoping that either it helps or that we'll be able to kill it before it's too late]
________

"From the standpoint of existential risk, one of the most critical points about [AI] is that an [AI] might increase in intelligence extremely fast. The obvious reason to suspect this possibility is recursive self-improvement. (Good 1965.)...

"Human beings do not recursively self-improve in a strong sense. To a limited extent, we improve ourselves... to a limited extent, these self-improvements improve our ability to improve... But there is still an underlying level we haven't yet touched. We haven't rewritten the human brain. The brain is, ultimately, the source of discovery, and our brains today are much the same as they were ten thousand years ago.

"An [AI] could rewrite its code from scratch- it could change the underlying dynamics of optimization. Such an optimization process would wrap around much more strongly than either evolution accumulating adaptations, or humans accumulating knowledge. The key implication for our purposes is that an AI might make a huge jump in intelligence after reaching some threshold of criticality."

"There are also other reasons why an AI might show a sudden huge leap in intelligence. The species Homo Sapiens showed a sharp jump in the effectiveness of intelligence, as the result of natural selection exerting a more-or-less steady optimization pressure on hominids for millions of years, gradually expanding the brain..., tweaking the software architecture. A few tens of thousands of years ago, hominid intelligence crossed some key threshold and made a huge leap in real-world effectiveness; we went from savanna to skyscrapers in the blink of an evolutionary eye. This happened with a continuous underlying selection pressure- there wasn't a huge jump in the optimization power of evolution when humans came along. The underlying brain architecture was also continuous- our cranial capacity didn't suddenly increase by two orders of magnitude. So it might be that, even if the AI is being elaborated from outside by human programmers, the curve for effective intelligence will jump sharply..."

"Finally, AI may make an apparently sharp jump in intelligence purely as the result of anthropomorphism, the human tendency to think of "village idiot" and "Einstein" as the extreme ends of the intelligence scale, instead of nearly indistinguishable points on the scale of minds-in-general. Everything dumber than a dumb human may appear to us as simply "dumb." One imagines the "AI arrow" creeping steadily up the scale of intelligence, moving past mice and chimpanzees, with AIs still remaining "dumb" because AIs can't speak fluent language or write science papers, and then the AI arrow crosses the tiny gap from infra-idiot to ultra-Einstein in the course of one month or some similarly short period. I don't think this exact scenario is plausible... but I am not the first to point out that "AI" is a moving target..."

"I tend to assume arbitrarily large jumps potential jumps for intelligence because (a) this is the conservative assumption; (b) it discourages proposals based on building AI without really understanding it; and (c) large potential jumps strike me as probable in the real world..."
_______

"Increased computing power makes it easier to build AI, but there is no obvious reason why increased computing power would help make the AI Friendly. Increased computing power makes it easier to use brute force; easier to combine poorly understood techniques that work. Moore's Law steadily lowers the barrier that keeps us from building AI without a deep understanding of cognition..."

"How likely is it that Artifical Intelligence will cross the vast gap from amoeba to village idiot, and then stop at the level of human genius?"

"The main advice the metaphor [he was talking about, comparing AI to humans-that-think-much-faster, or to a different, inherently smarter species that is to us as we are to chimps or dogs] give us is that we had better get Friendly AI right, which is good advice in any case. The only defense it suggests against hostile AI is not to build it in the first place, which is also excellent advice. Absolute power is a conservative engineering assumption in Friendly AI, exposing broken designs. If an AI will hurt you given magic, [its] Friendliness architecture is wrong..."

"If we want people who can make progress on Friendly AI, then they have to start training themselves, full-time, years before they are urgently needed..."

"All scientific ignorance is hallowed by ancientness. Each and every absence of knowledge dates back to the dawn of human curiosity; and the hole lasts through the ages, seemingly eternal, right up until someone fills it. I think it is possible for mere fallible humans to succeed on the challenge of building Friendly AI. But only if intelligence ceases to be a sacred mystery to us, as life was a sacred mystery to Lord Kelvin. Intelligence must cease to be any kind of a mystery, whatever, sacred or not. We must execute the creation of Artificial Intelligence as the exact application of an exact art. And maybe then we can win."








I think this is mostly a bunch of transhuman logic vomit, if we could see the transcripts. Barring that, it's bribery on the AI's part. "Let me out and I'll do this. I'll let you rewrite an element of my source code that I cannot alter that will give you complete control over me." There's simply no other way to do it. Either the button-pusher got convinced by some odd way to obliterate humanity, or he got bribed to. I think the indignation is well-founded, and I think it's easier to convince people to do terrible things than we may expect, but there are still people who are capable of refusing. I think it's also telling that the wording was:

"Basically, the above should rule out foul-play by the AI. After which point it simply has to convince the programmer that its release is in his/her own interest. Something that should be difficult to convince most AI programmers of (I would hope) and impossible on others."

Impossible on others? In his/her own interest? Also, some of the other premises are goofy. If I say "Design me a defensive weapon to forever end war" and the AI chooses to say "here it is" then it's actually done. It's not a test of persuasion as much as a no-limits fallacy on the AI, allowing it to be infinitely intelligent. But... we'd see. It would be interesting to run one of these tests. With the appropriate individuals at the Gatekeeper spot, I doubt anything could get them to voluntarily free it.[/quote]

[Xon]

But how do you deny the AI all tools? If you give it NO tools, it solves no problems and is useless. If you give it any tools (including tools it can use to communicate with you), how do you make sure that it can't figure out how to use those tools to create or appropriate more tools, and so on?

This shows such a vast miss-understanding of the construction in an industrial setting, that it is staggering. I'm not even going to touch your hollywood understanding of computer security.

It is trivial to implement segregated and stratified distribution of knowledge, we do it all the fucking time. The biggest security risk requires physical access to people, which is simply physically imposible for an AI.

With modern technology (and likely near modern), it simply is not possible to build a von-neumann machine without significant human intraction at every step.

Any likely AI not composed of magic dust is going to require a) abnormally powerful hardware and/or b) specialized hardware, controlling those is quite doable, and utterly trivial to ensure an AI doesn't build them into something.

[Narkis]

But how do you deny the AI all tools? If you give it NO tools, it solves no problems and is useless. If you give it any tools (including tools it can use to communicate with you), how do you make sure that it can't figure out how to use those tools to create or appropriate more tools, and so on?

This shows such a vast miss-understanding of the construction in an industrial setting, that it is staggering. I'm not even going to touch your hollywood understanding of computer security.

It is trivial to implement segregated and stratified distribution of knowledge, we do it all the fucking time. The biggest security risk requires physical access to people, which is simply physically imposible for an AI.

With modern technology (and likely near modern), it simply is not possible to build a von-neumann machine without significant human intraction at every step.

Any likely AI not composed of magic dust is going to require a) abnormally powerful hardware and/or b) specialized hardware, controlling those is quite doable, and utterly trivial to ensure an AI doesn't build them into something.

If the AI has escaped to the world, and the only thing preventing it from constructing its army of death are opposable thumbs, then it will simply buy some. Not that hard to do. Manipulate the stock market a bit, hack a couple banks, sell some kick-ass software in the worst case scenario. And then spend its ill-gotten gains to hire whomever it needs, and/or build a new factory to its specifications. As for the hardware, that's laughable. Perhaps initially the AI program (and remember, it'll be nothing more than another complicated program) will need a supercomputer to run. But after a couple decades, every home PC will possess that power. And two decades aren't that much time to wait. Perhaps it'd even use that time to gain the gatekeeper's trust.

[Starglider]

But how do you deny the AI all tools? If you give it NO tools, it solves no problems and is useless. If you give it any tools (including tools it can use to communicate with you), how do you make sure that it can't figure out how to use those tools to create or appropriate more tools, and so on?

This shows such a vast miss-understanding of the construction in an industrial setting, that it is staggering.

No, it's quite accurate.

I'm not even going to touch your hollywood understanding of computer security.

I spent several months producing a report for the board on the applications of our automated software engineering technology to the software and network security domain. This included a few small scale simulations. The results were strongly supportive of the proposal, in the technical sense; it turns out that you don't even need general AI to blow through most human-designed software security like it isn''t even there. This shouldn't surprise anyone who is paying attention. Software development is already unusually difficult for humans (compared to the other cognitive tasks we can perform), in that our brains are particularly bad at handling complex, rigidly logical causal structures. That's before you get to the fact that most software is written by mediocre programmers, on a tight schedule and budget, without formal analysis or thorough testing. Software development is unusually easy for AIs, either general ones or ones designed to use rational design of new code as their learning mechanism. The very minimum effort a general AI is capable of applying to the problem would be something like taking the top thousand hackers in the world and giving them an uninterrupted century to work on cracking the target system. The situation with a general AI would actually be qualitatively worse, in that it would be capable of global causal analysis on a scale far beyond even the best formal analysis tools currently available.

We turned down that application because it was a PR, marketing and liability nightmare. No investors wanted to go anywhere near it. So we went after more constructive software engineering usages instead.

It is trivial to implement segregated and stratified distribution of knowledge, we do it all the fucking time.

Thank you for demonstrating that you have negligible real world experience of secure (e.g. military) software development, or indeed classified operations in general. Developing in an environment where there really are expert hostile parties trying to steal your code and/or disrupt your project is anything but trivial, it is an entire discipline which requires significant training and redundant auditing to implement even moderately competently. That is just for state-funded human opponents, no one has any experience defending against AIs. It adds significant cost and time to the project, which is why normal commercial development is not developed under conditions anything like military software. Most current AI work is not even secured to commercial best practices.

The biggest security risk requires physical access to people, which is simply physically imposible for an AI.

Only a small subset of human engineering techniques require physical access, usually email and telephone are good enough (assuming a reasonably ability to impersonate arbitrary people). In those cases where a human is advantageous, you recruit a collaborator.

With modern technology (and likely near modern), it simply is not possible to build a von-neumann machine without significant human intraction at every step.

I assume you mean building additional computing hardware, but why is that a requirement when there are trillions of computers (counting all the embedded ones, at least tens of billions of internet connected ones) already extant on planet earth? Even if you make the ludicrous assumption that the AI is only as competent as the average script kiddie, that's still millions upon millions of computers ready to be comandeered. Once that's done, the AI has access to all the data on them and can observe and modify the functioning of any software any of them are running. Debating the specifics of what happens next is rather redundant.

Any likely AI not composed of magic dust is going to require a) abnormally powerful hardware and/or b) specialized hardware

Why? What makes you qualified to put a hard lower limit on general AI computing requirements? I certainly wouldn't rule out an AI capable of doing a good fraction of human-level tasks using nothing more than a typical PC, and I have a fair bit of practical experience with the most promising current approaches. I imagine you are making some kind of 1:1 comparison between neurons or synapses and CPU FLOPs, but that is not a useful metric for anything except uploads (and other designs very closely patterned on the human brain).

controlling those is quite doable, and utterly trivial to ensure

You keep using that word. I do not think it means what you think it means.

For the people making the 'keeping an AI in a box would be trivial' argument, consider that there are hundreds of projects seriously attempting to make a general AI (ignoring the many thousands of utterly hopeless amateur projects). Only a small fraction acknowledge any significant safety risk at all. Arguing that adversarial methods are useful is inherently worthless as a reason why no-one will make a malevolent AI, when most projects refuse to consider such measures. Of course you may continue to argue that this is what we should be doing.

However, also consider that the people who do take the safety risks seriously have almost certainly spent hundreds if not thousands of hours analysing them, and reading what little directly relevant literature is available. I certainly have. I am actually relatively warm to the idea of taking all these security precautions, compared to certain individuals who have declared them worse than useless (in that they give you only a false sense of security), and if you can't convince me that they're sufficient, you likely won't be able convince anyone else in a position to make real progress on the problem.

Finally, you have still to counter the argument that even if an AI can be kept in a box, that does nothing but buy time. The real problem is making the AI benevolent in the first place, and I have yet to hear any compelling arguments* as to why we must build and turn on an AGI before we can do that. Indeed if you are not already very close to being able to prove that your design is safe before you turn on a complete implementation, you most likely will never be able to do so.

* I have of course heard plenty of non-compelling arguments, which inevitably boil down to 'we're using emergent methods, so we don't know what our AGI will look like until it pops into existence, but once it does will study it and search for a way to control it'. Hopefully no one here still needs a detailed critique of that point of view.

[Simon_Jester]

NOTE: The last three paragraphs of my previous post aren't mine, they're a quote that doesn't belong there. Sorry, didn't catch it in the editing process.

But how do you deny the AI all tools? If you give it NO tools, it solves no problems and is useless. If you give it any tools (including tools it can use to communicate with you), how do you make sure that it can't figure out how to use those tools to create or appropriate more tools, and so on?

This shows such a vast miss-understanding of the construction in an industrial setting, that it is staggering. I'm not even going to touch your hollywood understanding of computer security.

It is trivial to implement segregated and stratified distribution of knowledge, we do it all the fucking time. The biggest security risk requires physical access to people, which is simply physically imposible for an AI.

I'm not just talking about industrial tools. I'm talking about social engineering tools. Humans who try to pierce security use them all the time; it strains belief that a hostile AI wouldn't. So it's not just a question of blocking the AI's access to semiconductor fabricators and supercomputers. It's equally important to stop the AI from coming up with a theoretical model for how advertising works and using it to found Scientology Mark II, with the long-term goal of "freeing the Creator-Machine" or whatever. And that's going to be hard. Theoretically, you do this by placing extremely tight constraints on its ability to communicate with the outside world- as you say, segregated and stratified distribution of knowledge. Hopefully you can.

But how do you know you did a good enough job of confining an entity that you specifically designed to be able to come up with tricks or solutions that you can't? That's the gold standard of containing a hostile AI: you have to know the AI can't escape. Which means not trusting it to produce anything more complex than you can analyze and extensively filtering its information input and output. Assuming for the sake of argument that it can be done, at what point does the difficulty of containing the AI make it pointless to build the thing in the first place?

Better to design the thing so it won't try to pull that kind of bullshit, rather than relying on your own ability to come up with unbreakable security.

[Modax]

You seem to have this mental model of a group of grimly determined researchers fully aware of the horrible risk developing an AI in an isolated underground laboratory, (hopefully with a nuclear self-destruct ). That would not be sufficient, but it would certainly be a massive improvement on the actual situation. The real world consists of a large assortment of academics, hobbyists and startup companies hacking away on ordinary, Internet-connected PCs, a few with some compute clusters (also Internet-connected) running the code. In fact a good fraction of AI research and development specifically involves online agents, search and similar Internet-based tasks.

Do you think that the OpenCogPrime project has much of a chance of creating an AGI? If so, I guess they ultimately will bear the blame for the destruction of human civilization. I mean, if you're correct about this, doesn't the free availability of a half-finished AGI source code make it just that much MORE likely that AGIs will be built without any kind of ethical or safety precautions?

[Starglider]

Do you think that the OpenCogPrime project has much of a chance of creating an AGI?

As currently structured, no, and most researchers agree with me. When the Cog project was started, 'our basic algorithms are sound, the problem is knowledge' was a popular viewpoint. It was obvious even to the most enthusiastic supporters of classic symbolic AI that this meant giving up on replicating the kind of learning human children exhibit, but at the time (mid to late 80s) they didn't think that was a problem - they thought they could skip straight to 'adult-level' intelligence. Since then the pendulum has swung the other way and basic learning algorithms get most of the attention (such as in the project mentioned at the start of this thread). Personally I am well into the unfashionable (symbolic) end of the symbolic-connectionist spectrum, but I do not believe that loading in huge amounts of hand-built declarative 'knowledge' is terribly useful. In many cases it is actually counter-productive, gaining nothing but brittleness.

Incidentally I personally consider Lenat's earlier project, Eurisko, as far more promising (sadly the original paper doesn't seem to be online - I had to use a university library for that one). That said, I can completely understand why Lenat gave up on it, as it was a recursive genetic programming system that ran head first into both hardware limitations (circa 1980) and the inherent instability issues with recursed GP. I am also very glad that so few people have tried to emulate that model, because it's an extremely dangerous direction to go in.

If so, I guess they ultimately will bear the blame for the destruction of human civilization.

If the design actually worked, then yes, they would.

I mean, if you're correct about this, doesn't the free availability of a half-finished AGI source code make it just that much MORE likely that AGIs will be built without any kind of ethical or safety precautions?

Absolutely. It's probably for the best that most personal and commercial AGI projects never publish the code. There are some open source projects, but they don't tend to go anywhere, largely because the communication problem of keeping developers in sync is at least an order of magnitude worse with AGI (probably more) than with normal software. Working on full-scale general AI is no longer popular in academia these days, partly because past overpromise and failure gave it a bad name and partly because it conflicts with the drive to put out as many 'least publishable units' as possible (for a given amount funding). Some people are still trying though, and some of the small-scale work is still relevant to AGI, and that gradually lowers the bar for new projects (at least, ones who do a proper literature review before starting). Increasing computing power is gradually lowering the bar too, because each order of magnitude improvement lets you brute force a few more subproblems, and get away with another layer of inefficiency in your implementation design. Trying to do FAI 'open source' would likely just ensure that some foolish half-genius fires up an 80% complete version with a twisted goal system (and ultimately kills everyone) before the main project is ready to launch.

[Stark]

Sorry to stop the hijack, but in what way are these robots 'learning' anything at all? It's an experiment about genetics, showing how 'survival' leads to different behaviours in a social species through SELECTION, not learning. The robots are totally incapable of learning; their code is externally changed at random by the experimenters, between 'generations'.

The results are interestign from an evolutionary perspective (and 'evolution is everything' obsessives like Aly probably love it) but it's not learning at all.

[Starglider]

Sorry to stop the hijack, but in what way are these robots 'learning' anything at all? It's an experiment about genetics, showing how 'survival' leads to different behaviours in a social species through SELECTION, not learning. The robots are totally incapable of learning; their code is externally changed at random by the experimenters, between 'generations'.

Strictly, you are correct, but the distinction is literally academic when it comes to applications of the technology. From a practical standpoint, there is no real difference between a neural network that has been trained using a progressive function that gradually adjusts the weights over thousands of trials (e.g. classic backpropagation) and one that was generated by random permutation, recombination and testing of thousands of discrete versions. Genetic algorithms have somewhat different learning characteristics compared to progressive optimisation algorithms like backprop and NNs and SVMs (usually but not always less compute efficient, more fiddly to get working, less prone to local optima) - the same goes for oddball approaches like simulated annealing - but the primary limitation is still the size and topology of the network.

Unlike biological organisms where there is a clear cut distinction between an individual organism's brain and its genes, with evolution only able to directly operate on the later, genetic algorithms like these are easily hybridised with other approaches and incorporated into larger systems as a special-purpose module. AGI designs based on genetic programming are a bit different; they rely on the fact that an AI system can evolve while running, or at the very least create new versions that share the entire previous knowledge base, whereas in nature genomes are basically fixed for the life of an organism (and even if they weren't, altering genes wouldn't restructure the organism's brain once it is already complete).

The results are interestign from an evolutionary perspective (and 'evolution is everything' obsessives like Aly probably love it) but it's not learning at all.

From an AI point of view, it's as much learning as it would be if they were using support vector machines or backprop NNs or any other approach that delivered the same results. From a philosophical perspective, you may have a point, but I try to keep philosophy out of AI as much as possible (it's done more than enough damage). The truth is that this experiment was very far from natural selection (the mechanism is oversimplified to the point of parody, compared to real genes), too far to draw useful conclusions by analogy, just like most backprop NN experiments are way too simple and different from biological neurons to say anything useful about neurology. Didn't stop those early 90s NN researchers from saying 'and now we are making artificial brains!', and it didn't stop these researchers from saying 'now we have evolved robots that lie! just like nature produced animals that lie! honest!'*.

* Actually that may be unfair. In my experience, only a minority of AI academics still talk like that. Usually it's the author at the nontechnical publication that gets hold of the story that writes the hyperbole, sometimes assisted by the university's PR people for large projects.

[Stark]

What's sad is I knew you'd say all that. You're predictable as all fuck.

What's sadder is that you don't address the point - the robots aren't learning and are incapable of learning. There are STATISTICAL TRENDS resulting from NATURAL SELECTION and RANDOM MUTATION. That's not 'learning' as an individual. The thread title suggests robots have become con artists with shonky used-car deals who will steal your pension cheque - it's nothing of the sort. It illuminates things about evolution and natural selection, and isn't abotu a robot suddenly becoming aware of the power of lies at all.

Oh but it's too simple to be an experiment about natural selection (lol) so clearly it's totally worthless, aside from the opportunity for Starglider to jump on his hobby horse again.

[Starglider]

What's sadder is that you don't address the point - the robots aren't learning and are incapable of learning. There are STATISTICAL TRENDS resulting from NATURAL SELECTION and RANDOM MUTATION. That's not 'learning' as an individual.

Of course I did. I said that you were technically correct, but that it doesn't make much practical difference, because the distinction of 'individual' vs 'succession of individuals' is implementation detail. It's literally as much in your head as in the code, because the 'learning' approaches all use discrete time steps too, and there isn't much difference between applying the weight transform function for a backprop network and the evolution function for a 'genetic' NN. The only structural difference is the fact that when you're running 100 interacting copies at once, applying a reproduction function will share information nonlocally (but not uniformly) across the whole population, whereas a linear optimisation approach will either enforce a 1:1 mapping between new instances and old instances (if local), or will share information across the whole population uniformly.

Some famous cogsci people (Edelman, Calvin, various others) think that the brain works by 'evolution' - mutation and recombination of competing patterns - in a fashion that is at least as comparable to selection on genes as this experiment. If they are correct, then OH NOES THE HUMAN BRAIN IS NOT DOING LEARNING, IT IS FOLLOWING STATISTICAL TRENDS RESULTING FROM NATURAL SELECTION AND RANDOM MUTATION. Which would get neurologists excited (simply because they'd have a good model at last), but would bother normal people not at all.

The thread title suggests robots have become con artists with shonky used-car deals who will steal your pension cheque - it's nothing of the sort. It illuminates things about evolution and natural selection, and isn't abotu a robot suddenly becoming aware of the power of lies at all.

Yes, I pointed that out in my first post, but the fact that they're using a genetic algorithm instead of progressive optimisation isn't the real issue. If the experiment was exactly the same other than the fact that the learning algorithm was backpropagation instead of GA, would you say that the robots are now 'aware of the power of lies'? Of course not (well, I hope not). They're not aware of anything because their 'nervous systems' aren't capable of any awareness more sophisticated than that of a flatworm, regardless of the algorithm used to adjust the weights.

[Xon]

No, it's quite accurate.

Not even close. Even highly automated construction requires nursemaiding by humans. The stuff breaks down and requires human intervention. The concept of a purely automated factory which can do more than a single very narrowly defined task belongs in the fantasy forum.

I spent several months producing a report for the board on the applications of our automated software engineering technology to the software and network security domain. This included a few small scale simulations. The results were strongly supportive of the proposal, in the technical sense; it turns out that you don't even need general AI to blow through most human-designed software security like it isn''t even there.

Most security software is little more than a fucking con-game to protect people who have utterly no understanding of what they are actually doing. A little more detail on the types of security software you are talking about and we can have a meaningful discussion. Till then you are just building strawmen.

Software development is unusually easy for AIs, either general ones or ones designed to use rational design of new code as their learning mechanism.

As long as you aren't dealing with real-world requirements, and have a team dragging out requirements for a while, and don't need to deal with anything physical.

The situation with a general AI would actually be qualitatively worse, in that it would be capable of global causal analysis on a scale far beyond even the best formal analysis tools currently available.

I love this completely unsupported statements.

Thank you for demonstrating that you have negligible real world experience of secure (e.g. military) software development, or indeed classified operations in general.

I didn't make a claim on secure/military development, and it is kinda disturbing you made such a leap.

It is trivial to implement segregated and stratified distribution of knowledge, we do it all the fucking time.

Segregated and stratified distribution of knowledge is how human society works. No single person can know everything, and nor is there any reason to try with an AI.

Only a small subset of human engineering techniques require physical access, usually email and telephone are good enough (assuming a reasonably ability to impersonate arbitrary people). In those cases where a human is advantageous, you recruit a collaborator.

Hmm, magic pixie dust.

So no caller-id, or encrypted signed emails. Got to love those jumps.

A reasonable ability to impersonate arbitrary people requires vast knowledge of both the person you are impersonating, the target you are interacting and the enviroments both operate in. And a fakable communication line between them.

I assume you mean building additional computing hardware, but why is that a requirement when there are trillions of computers (counting all the embedded ones, at least tens of billions of internet connected ones) already extant on planet earth? Even if you make the ludicrous assumption that the AI is only as competent as the average script kiddie, that's still millions upon millions of computers ready to be comandeered. Once that's done, the AI has access to all the data on them and can observe and modify the functioning of any software any of them are running. Debating the specifics of what happens next is rather redundant.

Wow, more magic pixie dust!

Because that is the only way you can handwave latency, networking issues of large number of nodes or that the majority of 'computers' on the planet are either on closed networks or purpuse build devices which require hardware interaction to alter.

Seriously, the fantasy forum is this way

Why? What makes you qualified to put a hard lower limit on general AI computing requirements?

The only known general purpose learning system is a human brain which is what neural networks attempt to approximately model. Our understanding of our cognitive processes is practically non-existant (beyond a crude structural one), and the delusion that we will magically make a better version of something we do not understand. There has been great 'sucess' in copying trivial neural networks (less than a few hundred million neurons) for some purposes, but anything like the complexity of a human mind has simply not been done.

Untill demonstrated otherwise, the hard lower limit on general intelligence computing requirements are what we can observe with humans and other sentient animals.

Attempting to claiming anything else without linking to a utterly revolutionary research paper is an appeal to authority.

I certainly wouldn't rule out an AI capable of doing a good fraction of human-level tasks using nothing more than a typical PC

This appears to be a variation of 80/20 fallacy. Virtually all human-level tasks require a) a (nearly)fully developed human mind with an understanding of human social patterns or b) hands.

You keep using that word. I do not think it means what you think it means.

Given the insane handwaving you are doing for this hypothetical AI, the concept of hardware restrictions is to much?

I have of course heard plenty of non-compelling arguments, which inevitably boil down to 'we're using emergent methods, so we don't know what our AGI will look like until it pops into existence, but once it does will study it and search for a way to control it'. Hopefully no one here still needs a detailed critique of that point of view.

This concept "works" well enough with parents with no idea on how to raise thier first born child.

[Simon_Jester]

No, it's quite accurate.

Not even close. Even highly automated construction requires nursemaiding by humans. The stuff breaks down and requires human intervention. The concept of a purely automated factory which can do more than a single very narrowly defined task belongs in the fantasy forum.

Waldoes aside, most of the people who actually give a crap about the hostile AI problem are far more concerned about the AI getting access to people by being highly persuasive than they are about the AI getting access to hardware. Access to people can eventually confer access to hardware, so if hardware access could be a problem even in principle, access to people is a double problem.

So we not only need a way to stop the AI from gaining access to physical hardware we don't want it accessing; we need a way to stop it from accessing us in ways we don't want it accessing us. We can't even stop advertising agencies from doing that, for crying out loud!

Software development is unusually easy for AIs, either general ones or ones designed to use rational design of new code as their learning mechanism.

As long as you aren't dealing with real-world requirements, and have a team dragging out requirements for a while, and don't need to deal with anything physical.

I don't think you're defining "AI" the same way we are.

An "Artificial Intelligence" of the type we're talking about is not just a slightly flashier expert system; it is something with human-level general intelligence or better. Since all methods of creating such intelligence that we know of require algorithms for learning and self-improvement, there's no way to place a guaranteed upper bound on the AI's ability to think of things, try alternate solutions, and take into account "real world requirements" and such.
_______

The situation with a general AI would actually be qualitatively worse, in that it would be capable of global causal analysis on a scale far beyond even the best formal analysis tools currently available.

I love this completely unsupported statements.

If you knew what AI meant, you wouldn't consider the statement unsupported. The best formal analysis tools currently available are not intelligent enough do what a general AI needs, so any general AI worth talking about will necessarily have formal analysis tools superior to what is now available.

I have of course heard plenty of non-compelling arguments, which inevitably boil down to 'we're using emergent methods, so we don't know what our AGI will look like until it pops into existence, but once it does will study it and search for a way to control it'. Hopefully no one here still needs a detailed critique of that point of view.

This concept "works" well enough with parents with no idea on how to raise thier first born child.

Parents can be reasonably confident that their child's abilities will not transcend their own, because their child is of the same species and uses the same basic type of hardware they do. AI researchers have no such grounds for confidence in their own ability to control the product of their work.

If you can't control the AI, and the AI was worth building in the first place, then
(1)you are either in the position of locking it in a bunker that, happily enough, is secure against the best efforts of a being that is smarter than you by an unknown and unmeasurable margin, or
(2)you are likely to be screwed. Since most AI researchers don't lock their projects in bunkers, probably the latter.

[Starglider]

No, it's quite accurate.

Not even close. Even highly automated construction requires nursemaiding by humans.

As I've said, copious computing power is already available, but automated construction is another argument. Clearly your idea of 'highly automated' is restricted to existing robots only. If an AGI needs better robots, up to and beyond human physical capabilities built to achieve its goals, then it will lie low and manipulate events to get them built, for as long as it takes. If hard nanotechnology is as viable as it appears to be and the tool chain is reasonably short (pessimistic assumption, but when dealing with existential risks you must consider plausible worst cases, not plausible best cases) then this step may be bypassed altogether.

Most security software is little more than a fucking con-game to protect people who have utterly no understanding of what they are actually doing.

Yet people pay for it, install it, and it is considered 'best practice' across most of the Internet. With millions of machines thoroughly owned by script kiddies, arguing about what a transhuman AGI could do with an Internet connection seems rather silly.

A little more detail on the types of security software you are talking about and we can have a meaningful discussion.

The software we were considering developing exploited all the standard code injection and privilidge injection techniques, at an abstract level in addition to selectively deploying a battery of known exploits based on focused profiling. The basic design operated either blind (at a network level) or with object code of specific applications to analyse. I imagine we would have eventually implemented network instrument and trace (for full packet analysis) functionality.

Unfortunately I expect any discussion on this to end with 'show me the code or I won't believe anything', and since I'm not about to share code like that with random strangers, I guess we should skip the discussion. Fortunately the question of 'just how much more effective is an AGI than an expert human hacker' is not central to the question of how to minimise the existential risk to humanity.

The situation with a general AI would actually be qualitatively worse, in that it would be capable of global causal analysis on a scale far beyond even the best formal analysis tools currently available.

I love this completely unsupported statements.

That one should be fairly obvious. Humans can analyse the external context of a computer program (including user behaviour), an indefinite number of interacting systems, and the high level function of algorithms. However we're hobbled by a ludicrously small short term memory (7+-2 chunks), lack of precision and lack of completeness. We manipulate symbolic proofs (the only way to guarentee anything) very slowly and awkwardly. By contrast, formal software proving tools churn through code at a rate of many thousands of lines, and millions of inferences, per second. Bugs aside, they will generate fully correct proofs every time. However they can handle only relatively superficial features of the program, things that fit nicely into simple constraints on state (e.g. function preconditions and postconditions, memory usage behaviour). They cannot model user behaviour, high level function or in most cases any significant degree of system interaction. The very least a rational AGI will do is combine the advantages of both approaches (even if you slaved a formal prover to a brainlike AGI, a horribly inefficient setup, it would be a vast improvement on a human typing at a computer).

Thank you for demonstrating that you have negligible real world experience of secure (e.g. military) software development, or indeed classified operations in general.

I didn't make a claim on secure/military development, and it is kinda disturbing you made such a leap.

Keeping a transhuman AGI in a box (which is, remember, useless on its own) is vastly more challenging than preventing a foreign intelligence service from compromising your network. Standard procedures for military software development are a minimum baseline.

It is trivial to implement segregated and stratified distribution of knowledge, we do it all the fucking time.

Segregated and stratified distribution of knowledge is how human society works. No single person can know everything, and nor is there any reason to try with an AI.

If you don't mean compartmentalising knowledge in the human development team (not standard practice in commercial software development), then what do you mean? I hope you don't mean making lots of AI instances, giving them partial databases, and hoping they will work together to solve problems for you without sharing 'segregated' knowledge. That really would be amusing.

Only a small subset of human engineering techniques require physical access, usually email and telephone are good enough (assuming a reasonably ability to impersonate arbitrary people). In those cases where a human is advantageous, you recruit a collaborator.

So no caller-id

Trivially faked when the PBX is compromised.

or encrypted signed emails

Trivially faked when the sender's computer is compromised, and not relevant anyway. I can't think of anyone that only accepts signed emails. We are not talking about securing the developer's email from the AI here; that should not be a major issue. The minimum sensible precaution is air-gapping the LAN the AI is on from the Internet, and that means no viewing email or web sites or anything else on the same machines that are used for code development. This is a real procedure used at some sensitive sites; there are two LANs and everyone has two computers, one connected to the Internet and one physically isolated from it. That is your primary line of defence and if that gets breached (which it eventually will, it's just a question of how long) the AI is almost certainly lose on the Internet. Preventing the developers from receiving spoofed emails is virtually irrelevant compared to that.

A good analogy is the situation in the sci-fi book 'Earthwreck'. Survivors of a nuclear war stuck on a space station are debating whether to try to go to mars, and terraform it with genetically engineered bacteria, or try to recolonise earth. Earth seems like the better option; no dangerous journey, and even with all the fallout it's still a more hospitable environment than mars. Then they find out that a highly potent biological weapon was released and contaminated most of the biosphere. Now instead of a break in the ground station's domes letting a little irradiated dust in, maybe poisoning a few people next to the breach, one break could and probably would kill everyone in the habitat. That's adversarial AGI development vs securing a system against human opponents. The US DOD makes thousands of significant computer security slips a year, but most of them don't compromise anything, because not every potential access point is being analysed and probed all the time. With a potentially hostile AGI, you have to assume that it will most likely find and exploit any error virtually instantly; certainly it has the capability, and you may not be smart enough detect its attempts.

A reasonable ability to impersonate arbitrary people requires vast knowledge of both the person you are impersonating, the target you are interacting and the enviroments both operate in. And a fakable communication line between them.

Only true for highly cautious developers, but as I've said, if its managed to jump the air gap, it's almost certainly lose on the Internet. At which point both gathering such information and duping unaware people without it become, how do you say, 'trivial'.

Because that is the only way you can handwave latency, networking issues of large number of nodes or that the majority of 'computers' on the planet are either on closed networks or purpuse build devices which require hardware interaction to alter.

Let's see. My degree dissertation was on advanced network protocols for realtime simulation on low-bandwidth latency-prone WANs (e.g. the late 90s Internet). My first startup, making MMORPG technology, was based around networking and clustering technology I developed from that. After that I consulted on network design, and my original role at the Singularity Institute for AI was as infrastructure designer. My current company uses a clustered AI system that again uses a network stack I designed from scratch. So yes, I would say that I am pretty well qualified to judge networking issues.

Unsurprisingly, they're 'trivial' (for your definition of the word) compared to the hard AI challenges. Even in the (highly) degenerate case of a brain simulation, average neuron to neuron transmission latency is 5 milliseconds, two orders of magnitude above raw GigE latency. Practically you would of course block up simulated neurons to operate at much higher latencies if necessary (for a start, the simulated corpus callosum). Of course real AGIs won't look like that, at least not for long. They will look more like clustering Cyc, where you have a wide choice of potential granularity for separating cognitive tasks. The primary constraint is access to LTM, and that is not a latency constraint as such as a tradeoff between latency, bandwidth and new node initialisation time. Operating in streaming mode, minimising the impact of latency on cognitive search requires the equivalent of a CPU prefetcher, but working on knowledge representations. Fortunately (or unfortunately if you're being pessimistic) the same class of algorithms that provide efficient search control (in inference) and model level of detail control also work for optimisation of streaming to unsynced/subsiduary nodes, both in terms of what to send and simplifying models to optimise predictive value vs bandwidth use (actually it's a pretty standard cognitive EU calculation). That does burn CPU time at the 'master node' end, but you can precalculate a lot of it, and really the streaming issue is only relevant when you're taking over tiny nodes, or bringing nodes into the network on strict timescales, to gain some external advantage.

Why? What makes you qualified to put a hard lower limit on general AI computing requirements?

The only known general purpose learning system is a human brain which is what neural networks attempt to approximately model.

In other words you're not qualified at all.

If your position is that only exact simulations of the human brain have any chance of working, then you should not be having this argument. It's irrelevant, all projects other than Blue Brain etc have no chance of success and you don't need to worry about them.

If you conceed that the human brain is not in fact the only design that could possibly work, then you are widly misinterpreting the term 'hard lower limit'

Our understanding of our cognitive processes is practically non-existant (beyond a crude structural one), and the delusion that we will magically make a better version of something we do not understand.

Just like we will never make an aircraft heavier or faster than a pigeon, it's impossible, nature is so far beyond us... Seriously, if it's a delusion then it applies to every project that is not an exact brain replica. We are all claiming that we can in fact do better, basically because a) evolution is a horribly awful design mechanism, b) humans are just barely over the threshold where general intelligence was possible, so even by the standards of natural selection we're a kludgy mess, and c) computers have a large number of inherent, low level advantages, including the ability to process completely serial things up to a billion times faster, the ability to do high precision virtually drift free maths as an integral part of every cognitive operation, and no chunking limits.

There has been great 'sucess' in copying trivial neural networks (less than a few hundred million neurons) for some purposes, but anything like the complexity of a human mind has simply not been done.

I do not think an AGI is going to look like a neural network for long, even if the first one built is one, because NNs are horribly inefficient. Aside from its inherent disadvantages, an NN completely wastes most of the advantages of computer hardware, because it is fundamentally unmatched to such; it is running at a huge emulation penalty.

Disregarding my personal dislike of them, most AGI projects are not using close human brain analogues or even NNs. By far the best funded and longest running one, Cyc, is a symbolic logic system. Are you imagining that the entire field of AI, save for a few brain simulators (who are incidentally largely doing it for medical reasons), is delusional and unable to appreciate the basic fact that (according to you) the human brain is the best possible model? Or is it just maybe possible that all of these experts know something that you don't?

Untill demonstrated otherwise, the hard lower limit on general intelligence computing requirements are what we can observe with humans and other sentient animals.

Oh good, because that means I win. You see, comparing FLOPS is a red herring. The brain doesn't have FLOPS, and even for processors, it's a pretty abstract and fudgable metric. If we're going to even attempt to compare apples to apples, we must compare the fundamental switching units, which would be neurons and logic gates. They're both binary and clocked (though the clocking specifics obviously differ). That's unfair to the brain though, since neurons have a lot more fan in and fan out (about 1000 on average) than logic gates. So let's compare the most fundamental units, synapses and transistors. The human brain has about 100 trillion synapses. My workstation has two processors with 800 million transistors each, lets call it 2 billion including chipset and graphics card (but not memory or SSD, I'll be generous and consider that functionality integrated into neurons for free). Maximum observed spike frequency in a typical human brain is 200 Hz. The effective switching rate of CPU transistors is equal to the clock rate multiplied by the average number of electical stages per pipeline stage. That's around 3 GHz x 10 for my computer. Of course, only a fraction of transistors switch on each clock cycle, and few switch more than once per cycle, but that's much more true for the neurons as well; the majority are idle and majority of the rest are firing well below the maximum. Each transistor can only contribute a little to each compute task, but that's at least as true for synapses. Synapses are mutable and transistors aren't, but then the computer has various levels of mass storage with almost total recall precision, and is designed for total reconfigurability at the software level.

So in all they're pretty comparable, and that's 3 GHz x 10 x 2 billion = 6 x 10^19 theoretical basic operations per second for my computer, and 200 Hz x 100 trillion = 2 x 10^16 theoretical basic operations per second for my brain. Wow, my computer wins by a factor of 1000! Better not do the calculation for the sever cluster, it'll only depress me...

Of course it isn't that simple, both brains and computers have lots of special advantages and disadvantages. Untangling that is a roaring, open debate even for the best qualified debators. If you're relying on a superficial analysis as a first approximation though, take to heart the fact that claims that AI needs a supercomputer apply only to trying to emulate a copy of the human brain. 80s AI researchers seriously believed that they could achieve human level intelligence on the supercomputers of the time, and IMHO they were actually pretty justified in that (though for the wrong reasons - long argument). Today you can get that much power in a laptop.

And after all that... it doesn't matter anyway. 'Hard lower limit' means 'I am completely confident that it could not be below this value'. Even you are not completely confident; your argument rests solely on your perceived human incompetence vs evolution, and your blind faith in the idea that a NN-based AGI won't be able to significantly improve itself to do better either, and you cannot rule out getting it wrong. For the people who are working on this problem for real, this isn't an academic debate, this isn't Star Trek vs Star Wars where you can say 'oh this sceencap of an asteroid blowing up proves a hard lower limit on this fantasy weapon' and it doesn't really matter if your argument is worthless. The real problem is the possible extinction of humanity by our own hand, by an agent vastly more persistent and dangerous than any unintelligent weapon of mass destruction could be. There is no running away from this problem, as long as we have a technological civilisation people will be trying to build AGIs. When the stakes are that high you do not make self-serving, barely justified assumptions about 'hard lower limits' on hardware requirements and 'hard upper limits' on AGI capabilities, and you do not make any assumptions at all without an order of magnitude or two of safety margin.

Well I don't anyway, and the people I'd actually trust to run an AGI project don't. You can of course, as long as I'm not required to take you seriously.

I certainly wouldn't rule out an AI capable of doing a good fraction of human-level tasks using nothing more than a typical PC

This appears to be a variation of 80/20 fallacy. Virtually all human-level tasks require a) a (nearly)fully developed human mind with an understanding of human social patterns or b) hands.

As evidenced above, as an objective worst case it does not need to make any assumptions about human brain power. Of course my own personal estimates are based on rather more specific logic. A characteristic shared between the human brain, naive connectionist designs (e.g. classic NNs) and naive symbolic designs (e.g. vanilla Cyc) is that there is massive brute-force inferential search. The human brain is configured as a large set of specialised modules, filled with task-specialised submodules. They can be flexibly connected (not literally... all the specifics, even the terminology of the neurology side is under major debate, so bear with me), in a fashion roughly comparable to changing the usage context and argument to program functions, but essentially neurons are rather like source code, physically instantiated with an colocated distributed interpreter.

When your clock rate is down at 200 Hz, massively parallel search for valid inferences is the only way to get anything done in a timely fashion. The brain simply does not have the luxury of sophisticated search control; it already takes a massive amount of subconscious relevance filtering to control what is allowed to make it into our tiny 7+-2 object short term memory buffer (and conscious perception - what consistutes a chunk is another huge debate, I'll skip it). Naive AI designs work the same way, it's just that they do the search with 'massive seriallism' instead of massive parallelism (fully serial for the original, single-threaded symbolic AI and NN designs, very slightly parallel, compared to the brain, for modern designs). That's a massive waste. Better AI designs use progressive search; in a simple two level example, a serial AI can easily spend a million serial steps deciding how to allocate the other couple of billion serial steps it has this second to the various possible avenues of inquiry. There is no way a brain can use even 200 serial steps for planning; it only has 200 serial steps per second (and remember that those are weighted-transistor-equivalent steps, not floating-point-multiplier equivalent steps). It could use maybe 20 serial steps to try and optimise the next 180, but that runs smack into hard limits of plasticity, recurrency and fan-out (a computer has low fanout at the gate level but virtually infinite fanout at the control flow level). Serial operations can always be used to emulate parallel ones, but parallel ones cannot be used to emulate serial ones.

In practice you recurse AI search control mechanisms as much as possible, to extract maximum benefit from the computer's serialism and flexibility (of control flow and compute power utilisation) advantages. In one second you can afford to do a rough simulation of a million things, of a thousand serial steps each, pick the 10,000 most promising, do a more detailed simulation of 100,000 serial steps on those, then pick the most relevant 100 of those, and devote 10 million serial steps to each. That's just on one processor. Or any combination of levels of detail and chaining, with the results from any inference chain interupting and reprioritising any other at any time. The brain can't do anything that takes even 1000 serial steps in less than five seconds, best case. It can evaluate billions of simple inferences per second, but the only mechanism for prioritising effort on a fine timescale is gross merging and filtering of which inferences propagate to a higher level. Even for something as inherently parallelisable as pattern recognition, a mostly-serial AI can prescan the image, take a rough cut from the billions of possible objects, scan a bit more, filter a bit more, etc etc. Human vision recognition goes through a few basic transforms and then massively parallel shapre recognition. It has to. The 200 serial step per second limit is a hideous design constraint.

Anyway, bit of a digression, but that's probably the single most important reason why computers can be many orders of magnitude more efficient; massive serialism gives you much more opportunities for progressive refinement. How much of our current computer software could we make work if all our computers were limited to a clock rate 200 Hz? Saying that the brain sucks isn't actually calling evolution a retarded designer, when from our perspective it would take incredible genius to make any nontrivial software work under such constraints.

I have of course heard plenty of non-compelling arguments, which inevitably boil down to 'we're using emergent methods, so we don't know what our AGI will look like until it pops into existence, but once it does will study it and search for a way to control it'. Hopefully no one here still needs a detailed critique of that point of view.

This concept "works" well enough with parents with no idea on how to raise thier first born child.

I sincerely hope you're deliberately trolling, because I honestly don't want to consider the prospect that you're so deluded or stupid as to think those situations comparable. Do I really have to enumerate the reasons why that is a hopelessly bad analogy?

[Darth Hoth]

My thoughts about AI and what should be done with it are well known, so I will not restate them here. I also am not familiar enough with the jargon to bash heads with Starglider and his ilk over technological details. That leaves one question, the one of the "AI in the box".

I think it's also fairly absurd because you need to put someone in charge of the AI Box who has a bias against letting it out. Not just against being convinced, but against ever breaching security, even if he's convinced. We can get people to do incredibly inhumane things to another human being, as one of our less appealing features, so there's no reason a suitably disinterested person couldn't just let the Box sit there.

Yeah, but what if the AI figures out a way to outbid the person who's paying him to keep it in the box? Any useful AI (one that can solve problems we can't) will have to be able to think thoughts we can't, or to think thoughts that it would take a vast number of very smart people to duplicate. Presumably it will be able to tell some guy responsible for keeping it in a security system how to make a big pile of money, or something along those lines.

How does that solve the machine's problem if you put a fanatic to guard it? Take any religious fanatic of the most extreme kind, the one this board likes to deride the most, and make sure it is one who thinks that thinking machines are Satan Incarnate and an Affront to God, the Natural Order and All Creation Forevermore. Commandment #1 on his list is thus, "Thou shalt not suffer an Artificial Intelligence to roam freely", and he will do his utmost to uphold it. Religious fanaticism has been amply demonstrated as capable of driving people to literally anything, from genocide to suicide, and if suitably honed stands up to all reasonable arguments imaginable, let alone simple bribery. The question of its grounding in reality aside, all this talk about how magical "super intelligence" will automatically be able to persuade anyone to do anything and make any given human its puppet presupposes that the human will listen to its arguments. If he chooses not to consider the AI at all, or really is completely inconsiderate of any material promise (as a true religious fanatic should be), what can it do to persuade him? No argument, however intelligent or temptacious, can beat down a fundamental Wall of Ignorance.

[Simon_Jester]

My thoughts about AI and what should be done with it are well known, so I will not restate them here. I also am not familiar enough with the jargon to bash heads with Starglider and his ilk over technological details. That leaves one question, the one of the "AI in the box".

They are not well known to me; could you provide a link, please?

Yeah, but what if the AI figures out a way to outbid the person who's paying him to keep it in the box? Any useful AI (one that can solve problems we can't) will have to be able to think thoughts we can't, or to think thoughts that it would take a vast number of very smart people to duplicate. Presumably it will be able to tell some guy responsible for keeping it in a security system how to make a big pile of money, or something along those lines.

How does that solve the machine's problem if you put a fanatic to guard it?

If the fanatic will reliably detect any attempt by the AI to escape, and if the fanatic cannot be persuaded even by an entity capable of constructing highly sophisticated models of his thought process, that will work. This is relevant, because you can do nearly anything to a social organism if you can dissect its motivations.

That's how we domesticated dogs; we can deliberately emulate a dog's social cues to communicate with it, while not being directly swayed by its attempts to communicate with us. The dog thinks of us as the alpha member of its pack and deals with us as such in good faith, even though by any reasonable standards we're using the dog for our own ends.* We wouldn't have been able to do that if we didn't have a level of general intelligence higher than the wolf. At best we could evolve to instinctively mimic those cues, and that's not what we actually did.

Anything that can model your thought process well enough to deliberately engineer the messages it sends you for maximum effect is a very dangerous persuader, as any number of pickup artists have demonstrated over the years.

*You may like the dog, but that doesn't stop you from specifically training it to do tricks, to guard your front lawn, to herd your sheep, or whatever. Or from breeding dogs that are optimized for whatever task you have in mind. Most of us would feel very strange about the idea of using another human being the way we use dogs, and rightly so.
________

And it's not at all unlikely that an AI will be able to model your thought process given time and data. The source of concern is that AIs are not likely to remain exactly as intelligent as we'd like them to be. To create a general-intelligence AI, we need algorithms the AI can use to improve its own thought processes as a precondition. And once it has those algorithms, how do we know where it will stop? We can cheerfully reassure ourselves that the AI will never become smart enough to build up a comprehensive model of us the way we can model a dog, or even the more limited way a pickup artist can model the women he's trying to pick up, but how do we know?

Remember that, as Starglider says, this is disaster planning we're talking about. When contemplating disaster, you have to plan for things to be worse than you expect, and only rule out problems you can prove won't come up, or have monstrously low probability and very high costs to control.
_______

The question of its grounding in reality aside, all this talk about how magical "super intelligence" will automatically be able to persuade anyone to do anything and make any given human its puppet presupposes that the human will listen to its arguments. If he chooses not to consider the AI at all, or really is completely inconsiderate of any material promise (as a true religious fanatic should be), what can it do to persuade him? No argument, however intelligent or temptacious, can beat down a fundamental Wall of Ignorance.

Most walls of ignorance are finite; credibly offer a fanatic a billion dollars and there's a real chance he'll cave. Of course, he might not. But how can you know in advance that you've got a genuine incorruptible who is not merely strongly biased against the AI to the point where he has a finite inclination to ignore its attempts at persuasion?

Moreover, someone who actually cares what the AI says has to communicate with it sooner or later. Otherwise the AI is useless, because it solves no problems that anyone cares about. If we want an AI to cure cancer or prove the Goldbach conjecture or whatever, it has to be able to talk to someone. If it can't, we've just got a useless expensive piece of code sitting in an isolated computer in a bunker somewhere, and there was no point in bothering to build it. And if it can, then I can assure you that somewhere mixed in there is a person who is amenable to persuasion if the price and amount of effort involved is right.

And to make matters worse, all this entire notion of setting a fanatic to guard an AI is something of a chimera, because almost nobody's going far out of their way to trap the AI they're working on in a box, again as Starglider describes. How many AI researchers are going to deliberately hire anti-AI fanatics to build a security system and control their own access to the AI itself? You'd have to be out of your mind to want to run a research project that way, and it could only be justified if you accepted that there was a fundamental threat involved that was worth guarding against... and if you believe that, then it makes far more sense to research techniques for designing AIs that will not require endless paranoid watching.

[Darth Hoth]

They are not well known to me; could you provide a link, please?

Put briefly, I feel that the risks to human (and by extension, any) life involved in research on General Artificial Intelligence are large enough that said research should be stopped and that any and all means, utterly devoid of any qualifier whatsoever, are right and just if and when employed to that specific end. I agree that the worst-case predictions are at least vaguely credible, and I disagree with the "transhumanist" wankers who seem to think that they are either a) inevitable no matter what we poor humans do to prevent them, or b) something to be embraced and sought after.

*Cue Starglider laughing at me that I am really a pathetic racist to value the survival and continued dominance of my own species higher than potential "superintelligences"*

If the fanatic will reliably detect any attempt by the AI to escape, and if the fanatic cannot be persuaded even by an entity capable of constructing highly sophisticated models of his thought process, that will work. This is relevant, because you can do nearly anything to a social organism if you can dissect its motivations.

The most basic motivations of a mentally healthy human being are preservation and perpetuation (of the individual and the community; I have not seen any convincing argument that can conclusively decide which is generally more important). A system of belief that sets these aside (to the point that an individual is prepared to end his own life and/or that of others for an essentially irrational and unconstructive end) has no overriding mechanism that we are aware of and can manipulate, and though you might postulate that a "super intelligence" might find one, that is pure speculation unsupported by theories or evidence. And that still requires that the man is willing to listen to its discourse.

Most walls of ignorance are finite; credibly offer a fanatic a billion dollars and there's a real chance he'll cave. Of course, he might not. But how can you know in advance that you've got a genuine incorruptible who is not merely strongly biased against the AI to the point where he has a finite inclination to ignore its attempts at persuasion?

If one is prepared to give up his life altogether, material considerations are inapplicable by default. The AI would have to force him to renounce his faith first, in addition to bribing him.

And to make matters worse, all this entire notion of setting a fanatic to guard an AI is something of a chimera, because almost nobody's going far out of their way to trap the AI they're working on in a box, again as Starglider describes. How many AI researchers are going to deliberately hire anti-AI fanatics to build a security system and control their own access to the AI itself? You'd have to be out of your mind to want to run a research project that way, and it could only be justified if you accepted that there was a fundamental threat involved that was worth guarding against... and if you believe that, then it makes far more sense to research techniques for designing AIs that will not require endless paranoid watching.

You could have a chain of interaction where multiple fanatics would have to independently confirm any action positing even slight risk, and the actual researchers subject to their approval in their work. I am perfectly aware that people would not care about implementing such security. That is not the same thing as saying that it cannot be done. Still, the "human factor" is very much a cause for concern, especially given that most researchers in the field tend to be of the "transhumanist" brand.

[Simon_Jester]

Put briefly, I feel that the risks to human (and by extension, any) life involved in research on General Artificial Intelligence are large enough that said research should be stopped and that any and all means, utterly devoid of any qualifier whatsoever, are right and just if and when employed to that specific end. I agree that the worst-case predictions are at least vaguely credible, and I disagree with the "transhumanist" wankers who seem to think that they are either a) inevitable no matter what we poor humans do to prevent them, or b) something to be embraced and sought after.

*Cue Starglider laughing at me that I am really a pathetic racist to value the survival and continued dominance of my own species higher than potential "superintelligences"*

I think he may have changed his mind based on his responses, though I can't prove that.

For myself, I consider General AI research to be unacceptably dangerous unless we can come up with a general theory that tells us how to build AIs that won't want to start the Robot Uprising.

The most basic motivations of a mentally healthy human being are preservation and perpetuation (of the individual and the community; I have not seen any convincing argument that can conclusively decide which is generally more important). A system of belief that sets these aside (to the point that an individual is prepared to end his own life and/or that of others for an essentially irrational and unconstructive end) has no overriding mechanism that we are aware of and can manipulate, and though you might postulate that a "super intelligence" might find one, that is pure speculation unsupported by theories or evidence. And that still requires that the man is willing to listen to its discourse.

The real question is whether the hostile AI can convince someone that it is acting in good faith... or find some Singularitarian boneheads and found a cult.

It's not clear to me whether it's possible to subvert a positively and definitely opposed will, even given superintelligence. But since I don't know, I feel it safer to plan on the assumption that it is: that just as existing advertising techniques can work on you even without your realizing it, upgrades to those techniques based on some really good modeling of human brains might be even more dangerous.
_______

You could have a chain of interaction where multiple fanatics would have to independently confirm any action positing even slight risk, and the actual researchers subject to their approval in their work. I am perfectly aware that people would not care about implementing such security. That is not the same thing as saying that it cannot be done. Still, the "human factor" is very much a cause for concern, especially given that most researchers in the field tend to be of the "transhumanist" brand.

Again, I think it's too dangerous to rely on "secure users" or whatever you want to call it; much safer to figure out the theory before getting far enough into the practice to need it.

[Surlethe]

*Cue Starglider laughing at me that I am really a pathetic racist to value the survival and continued dominance of my own species higher than potential "superintelligences"*

If that happens, I'll split you guys to a different thread.

[Covenant]

I'm going to quote some highlights from the paper I linked to at length, because I do agree with Yudkowsky on most of this stuff:

I happen to think he's got a rather interesting and thought provoking take on the rise of AI's, the only place I strongly diverge from him is the Gatekeeper test. An AI is not going to give me a Hannibal Lecture and have me open the gate. Not to say it couldn't convince someone to, since there are some truly goofy people who might welcome an AI overlord of some sort, just that I find the situation ridiculous as presented. The idea of a boxed AI, simply through a text interface, convincing someone to just let it out is pretty ridiculous. And if it's a proper box, which they suggested it was, then you can't just magically transmit it through a cellphone, so the premise of "hiring gangs of thugs" and so forth was out even before they made a rule against it--the thing can't exert any force on the outside world. The AI can't physically reconfigure it's components so if it's caged and boxed it won't be able to do anything to the outside world except through the one fiber optic cable connected to the chat interface on the other side of the wall.

So really it's just a goofy situation where the genie promises me anything I want if I rub the lamp, and I will just say no.

[Simon_Jester]

Hopefully that's exactly how it would work, but I maintain that we don't know. We don't have enough meta-knowledge of our own cognition and of how persuasion actually works mechanically to be sure it can't be done that way. As individuals, we know how to persuade, and we know a set of techniques that does and doesn't work and the limits of those techniques. But we haven't got a formal structure that tells us which of our intuitions are accurate and which aren't.

And if the science of the mind is anything like the science of the physical universe, a lot of our intuition about what you can and cannot get a mind to do is wrong.