Transhumanism: is it viable?

[LionElJonson]

The problem is not with out programming of a friendly AI. It is with the building of subsequent machines. First off, the machines will not be smarter than we are. It will have a faster CPU, the two are not the same thing. We have to program this thing to be able to do things like perform higher math and mimic having an actual biological intelligence. It will be limited in its ability to do these things by the human programmer who initially creates it and as a result will be limited in its understanding of higher math to the degree that the programmer understood higher math. In turn, it cannot subsequently program a machine that is smarter than it unless it uses some sort of genetic algorithm that creates random mutations in code and then selects for those machines which get built that display higher cognitive capacity. These same mutations however can create a non-friendly AI. As a result malevolent AIs could well (and eventually will, simply due to mutation and drift) evolve out of your recursively more intelligent AIs due to a lack of function mutation in the Friendly code.

This is easily demonstrated as false just by looking at the way humans learn higher math, we don't even need to bring computers into the equation. When discussing higher math with teachers in the subject, my mind does not run through countless attempts at trial and error and return with a solution that hopefully satisfies them. Invariably, if I am wrong, either I or they can actually pick out the logical error that was made.

You can learn higher maths this way because you have a teacher who knows his stuff to guide your way. In other words, the problem has already been solved, and your only task is to learn the material.

But the "trial and error" that is described by Alyrium is in regards to improvements on a design that the humans have already busted their balls programming to the best of their understanding. That's a horse of a different color. Apart from a minimal amount of guidance the humans may provide, the computer is all on its own and completely blind, and experience has shown the best way to solve these kinds of problems is trial and error.

Or, you know, taking what you already know and applying it. I'm pretty sure mathematicians aren't just blindly barrelling off into the darkness, you know.

Your claim amounts to 'rational AGI is impossible'. For rational AGI to be impossible, a complete algorithmic model of thought must also be impossible. Since every human being does it, claiming that something we do is impossible to fully understand is an extremely spurious statement.

Alyrium's statement is that 'rational AGI is probably impossible for us to realize'. The emphesised statement is important. We may simply not be smart enough to realize a rational AGI, even if it is possible in theory.

First off, we're not really rational ourselves. For the most part, our reasoning is post-hoc if it occurs at all. Only those of us who have trained our fundamentally flakey survival machine called a 'brain' to simulate rational thought can pretend to be rational. Even for them, their prejudices, ambitions, laziness, and so forth can get in the way to foul their reasoning. This is why we have the peer review process in the first place: to catch each other's mistakes.

Yes, and that's because our intelligence has a bunch of biases due to our evolution. An artificial intelligence probably would not possess them; if properly programmed it would be non-anthropomorphic and rational from the beginning.

The second point is that intelligence may not be a step-by-step list of instructions you can write down. There are many other types of algorithms that are not amenable to digital computing. We won't know for sure until intelligence is nailed down and understood.

Modern computing isn't a step-by-step list, either. That's what the "multi" in "multi-core computing" means.

It is certainly difficult. I think Starglider's numbers are off by a couple orders of magnitude. I think Kurzweil is rather optimistic about his timescales. But each year, programming techniques become more sophisticated, and writing self modifying and self analyzing code is becoming more the norm. If someone's off by a decade or two, well, predicting the future was never easy.

Predicting the future is almost impossible. Just ask any prophet.

Depends on the method you're using to predict the future. When you're shooting a rocket off, you can say with a good bit of certainty where it's going to be in six months' time.

Self-analysis is extremely limited, if it is applicable at all, due to Rice's theorem. There's no general algorithm to analyze a general algorithm to see if the partial function it implements has any non-trivial property, of which friendliness is one. This lies at the core of computational theory, and so its hard to dismiss with the assertion that "programming techniques improve every year." Programming techniques will not allow you to do the impossible no matter how sophisticated they are. We must first prove that friendliness is amenable to that kind of analysis.

Yes, there's no way to tell whether a particular algorithm is Friendly. You just have to design it with Best Practices and hope you've gotten it done before someone who isn't running a Friendly AI project finishes first.

And before you appeal to us as a friendly intelligence, realize that we do not fit our own definition of "friendly." We do immense damage to ourselves. We don't even know if a friendly intelligence can exist, let alone implement one.

Well, we know how to design an unFriendly AI; just upload a person, or run an evolutionary algorithm.

The second problem with this assertion is even if we prove that we can compute the answers, there's no guarantee to the complexity of the task. Many non-trivial problems explode violently in time and space requirements as the characteristic measures of its size increase, like the traveling salesman problem. Intelligence is intricate enough to expect that emulating it is going to take a fair chunk of computation, and analyzing it for properties like friendliness is going to increase that complexity to obscene values.

You don't analyze for Friendliness afterwards. You design for it from the get-go; it's a feature of goal structures and whatnot. We also know the bare minimum system requirements for intelligence: we've got a working example, sitting right behind our eyes.

Even if nanotechnology pans out, there will be no picotech revolution — once you get smaller than molecule size, there's hardly any structure to matter that one can organize into computing elements. This is neglecting the real challenges to nanotech computing, like the fact that molecular computing elements will be mechanically floppy and will not take much electrical disturbance before tearing themselves apart. The amount of computing horsepower availible will be limited no matter how you slice it.

Maybe, maybe not. AB-matter might or might not be possible; it's too soon to tell.

That would be nice. I don't know how development on those is coming along. Can you fill me in?

What is projected to occur in this scenario when robots become cheaper than wage slaves? Do you conjure that this development will be good for any of the workers who are being replaced? Do you believe that this transition will be easy, or peaceful? Do you believe that these replaced people will continue to eat? (Out of the goodness of capitalists' hearts, no doubt, that's gotten us so far.)

You seem to be assuming that greedy capitalists would be in charge of these robots.

RepRap finished Mendel the better part of a year ago: http://reprap.org
Fab@Home is still polishing up the Model 2: http://fabathome.org

The substance directly extruded by these methods are plastic. Do you really expect we will have no need for machined metal in the future?

Both websites read exactly like the overly-optimistic tripe that futurists often fall prey to. While home replication will have its place, it cannot replace industry. The materials we must work to achieve our high-tech society are too diverse for one small machine to be able to work them all. It cannot cure ceramics, it cannot cut steel rods, it cannot forge gears, and it cannot etch ICs. It cannot even work with all plastics. That is a very limited capability

It might be possible, as time goes on, for them to create designs with metal powders that are welded into place, or for nanotechnological solutions that manipulate individual atoms or molecules. I know for a fact that the medical community are building ones to print off organs, one layer of cells at a time.

The scenario parallels the early home computer industry, although it is proceeding at a slower pace (or so it feels like).

Yes, and look what happened to home computing. It fell by the wayside. Do not dismiss the power of the economy of scale.

To be honest, I fully expect something like these to be in every house; feedstock goes in, and food comes out. Remember what I said about post-scarcity? This is a part of the post-scarcity revolution. The part that stops riots over a lack of affordable food.

[Anguirus]

You mean, large movements with no central leadership can be widely varied and often internally divided?

The quotation you attempted to respond to carries therein a bigger issue than that. If self-identified transhumanists have no common set of beliefs, then all you're left with is self-identification.

Sadly, I won't have time for a Coliseum debate anytime soon, but I would follow one with avid interest.

I'm also extremely curious what your justification is for calling transhumanism a "large movement."

To be honest though I'm not sure exactly where I'd draw the line between transhuman application of technology and regular application of technology.

Yep, that's an interesting demarcation problem there.

Perhaps the best way I could put it offhand is that transhumanism involves actively making ourselves better in some way through technology, as opposed to simply using technology.

So...sanitation, medical science, electricity generation, the Green Revolution, transcontinental communication and transportation, would all of these still be considered "using technology" rather than "making ourselves better through technology?"

Are anabolic steroids on the "transhumanist" side?

Edit: would you say that isolating a number of genes that contribute to high and low intelligence and psychological problems and promoting the first while removing the second and third from the gene pool would be a concept which's eventual feasibility is "not supported"? I'd consider that at least borderline transhuman - it would result in a future humanity that would likely be noticeably different and more capable than the present one, albeit it would simply be a difference of changing the average rather than transcending the peak.

That sounds like eugenics. Good luck getting that to fly.

I'm not even trying to be snarky there. You vastly overestimate the societal will to accomplish such a thing. There are a pretty fair number of people who don't even believe in screening for Down syndrome.

And don't brush this off...the only difference between what you propose and eugenics is that instead of sterilizing people, you are aborting their fetuses whenever they manifest a "bad" gene. To be sure, this goes on to some extent today, but eradicating genes from the gene pool? Christ, we can't even get enough people to vaccinate their brats to get herd immunity.

We also have the same problem as all the rest of the vague stuff you're talking about...sure, we've found genes that are correlated with "the mental difference between humans and chimps" (I hesitate to even call that "genes for intelligence") as well as genes that are correlated with "schizophrenia", but that is astonishingly far away, in terms of our conception and understanding, from the kinds of things that you are talking about. (Alyrium got into this, but we are talking about vast regulatory networks, and these correlations I've mentioned are NOT 1:1, or even close.) Even gene therapy is just recently starting to be seriously talked about again, because ten years ago one of the first attempts killed a kid. (Funny, the effect that publicity can have on technology...where are all our nuclear power plants again?)

You're still talking about science fiction, rather than things that bioengineers are actually seriously talking about. If you wanted to do some good for your "cause," you should take a "transhuman" technology that actually exists (such as gene therapy, prenatal screenings, prostheses, sustainable power, vaccines) and find a way to advocate for it.

[Shroom Man 777]

You were watching Repo Men, weren't you, Shroom?

Or perhaps Repo! The Genetic Opera.

No, but I know of it. I'd love to watch it, it sounds absolutely nuts.

Still, I would love to see human beings, and their constituent organ systems, being trivialized and commercialized to the same extent as damn iPods and other crappy consumer products. It would be absolutely horrific, yet so wonderful.

[Junghalli]

So...sanitation, medical science, electricity generation, the Green Revolution, transcontinental communication and transportation, would all of these still be considered "using technology" rather than "making ourselves better through technology?"

They don't fundamentally improve our innate physical and mental capabilities beyond what nature would give our baseline selves growing up in a physically optimum* environment, so I'd say no.

* I wouldn't exactly call our present physical environment optimum for that matter, just look at the health problems caused by fatty modern diets, but it's closer to optimum than somebody living before those technologies would have experienced.

Are anabolic steroids on the "transhumanist" side?

Hmm, it's enhancing yourself above your baseline biological potential, I'd say yes, it's a crude form of transhumanism.

That sounds like eugenics. Good luck getting that to fly.

I'm not even trying to be snarky there. You vastly overestimate the societal will to accomplish such a thing. There are a pretty fair number of people who don't even believe in screening for Down syndrome.

Yes, I'm aware of that.

Transhuman technologies don't necessarily have to be universally adopted to potentially have a significant impact on civilization. If even a small percentage of the world's population did what I'm suggesting and it resulted in a noticeably lower rate of mental illness there's a positive effect on society right there. And that's not touching on the possibility that even a small increase in the number of smart people could potentially, say, noticeably increase the pace of technological advance by increasing the quantity of highly capable contributors.

You're still talking about science fiction, rather than things that bioengineers are actually seriously talking about.

Yes, and many technologies we take for granted would have been the stuff of science fiction 100 years ago. The way to turn them into the stuff of science fact is to invest in the necessary enabling technologies, e.g. investing in research into ways to make genome sequencing cheaper with an eye toward eventually getting it cheap enough that we can start sequencing the genomes of vast numbers of people and doing statistical analysis on them. Thankfully these enabling technologies are often things that have plenty of potential to improve human life in the here and now. We may invest in robotics with an eye toward eventual effective post-scarcity, but more autonomous robots have tons of much more immediate potential applications.

If you wanted to do some good for your "cause," you should take a "transhuman" technology that actually exists (such as gene therapy, prenatal screenings, prostheses, sustainable power, vaccines) and find a way to advocate for it.

To be honest at this point I don't really have the time, money, or inclination for serious advocacy, and even if I did I can think of more immediately demanding causes, like the rather depressing political situation in my country.

[Wyrm]

Or, you know, taking what you already know and applying it. I'm pretty sure mathematicians aren't just blindly barrelling off into the darkness, you know.

You'd be surprised how much tinkering is involved in a proof. The most famous proof of recent times, Andrew Wiles's proof of Fermat's Last Theorem (how gratifying it is to say that now without any appeals to tradition) involved a huge amount of tinkering to count the all the eliptic curves and the modular forms, and it was already known at the time that FLT was connected with the modularity theorem (then the Taniyama-Shimura-Weil conjecture), and you would get it for free with proving the TSW conjecture. It needed to be done anyway, because a large amount of work rested on that conjecture.

While you're right that the search isn't completely random, and that you can get a lot of mileage out of knowing what the rest of the field says about your subject matter, there is a large amount of semi-blind stumbling. And an amount of serendipity: Wiles had discounted a previous method of counting the two sets of objects because it was inadequeate to the task, only to realize that it neatly patched the holes of the method he first presented to the world.

Yes, and that's because our intelligence has a bunch of biases due to our evolution. An artificial intelligence probably would not possess them; if properly programmed it would be non-anthropomorphic and rational from the beginning.

Maybe, but it's hard to disentangle the biases from the intelligence. Remember that human rationality is sitting atop of a mound of instictual foundation. While we can characterize rational behavior, it's a lot harder to generate behavior that is both rational and useful from first principles.

The second point is that intelligence may not be a step-by-step list of instructions you can write down. There are many other types of algorithms that are not amenable to digital computing. We won't know for sure until intelligence is nailed down and understood.

Modern computing isn't a step-by-step list, either. That's what the "multi" in "multi-core computing" means.

Actually, that's wrong. There is nothing that can be programmed on a multi-core computer can be that cannot be done on a single-core computer with preemptive multitasking. It'll just take longer. If you take a look at a computer program, step by step instructions is precisely what it turns out to be. Indeed, making sure certain steps are completed in specific sequence is one of the bugbears of multithreaded programming.

I'm talking about things like differential equations when I refer to algorithms not amenable to digital computing.

Depends on the method you're using to predict the future. When you're shooting a rocket off, you can say with a good bit of certainty where it's going to be in six months' time.

But not with a system like the weather. Some systems are amenable like celestial mechanics, but even mildly chaotic systems like the weather will be essentially unpredictable even a week later. Human behavior is even more chaotic.

Yes, there's no way to tell whether a particular algorithm is Friendly. You just have to design it with Best Practices and hope you've gotten it done before someone who isn't running a Friendly AI project finishes first.

And will the self-modifying code always produce an algorithm that conforms to the Best Practices? And will the new improved self-modifying code also conform to Best Practices? Each step along the way introduces the possibility that the Best Practices will not be implemented in the modified child, and we're in big trouble.

Well, we know how to design an unFriendly AI; just upload a person, or run an evolutionary algorithm.

This is a peculiar meaning of the word "design" that I am not familiar with.

The second problem with this assertion is even if we prove that we can compute the answers, there's no guarantee to the complexity of the task. Many non-trivial problems explode violently in time and space requirements as the characteristic measures of its size increase, like the traveling salesman problem. Intelligence is intricate enough to expect that emulating it is going to take a fair chunk of computation, and analyzing it for properties like friendliness is going to increase that complexity to obscene values.

You don't analyze for Friendliness afterwards. You design for it from the get-go; it's a feature of goal structures and whatnot.

Same problem from the other end, and worse even. Instead of verifying that an algorithm always produces a friendly response, you are specifying an algorithm that produces algorithms that always produces friendly responses.

We also know the bare minimum system requirements for intelligence: we've got a working example, sitting right behind our eyes.

Except it's not friendly, and comes with all sorts of baggage as you have said previously. If we try to copy it without understanding it, we are only asking for trouble.

Maybe, maybe not. AB-matter might or might not be possible; it's too soon to tell.

That's one paper. Any others? We know that atoms spontaneously organize themselves into crystals. Nuclei don't spontaneously organize themselves into fibers. This is my one great difficulty with picotech. If their graph of the nuclear/electromagnetic interaction is correct (looking suspiciously like that of a normal electronic bond), then why don't nuclei bind naturally?

Sorry, but I have to see a lot more before I buy any femtotech argument. At least nanotech has a solid basis in fact behind it.

It might be possible, as time goes on, for them to create designs with metal powders that are welded into place, or for nanotechnological solutions that manipulate individual atoms or molecules. I know for a fact that the medical community are building ones to print off organs, one layer of cells at a time.

"Metal powder welded in place"? You call that a substitute for forging a metal piece? How do you ensure that all the crystals get heated evenly and fuse properly? How do you ensure that impurities don't sneak in? And how do you apply heat or work treatment to it?

And nanotech solutions? Mike puts it better than I can hope to.

To be honest, I fully expect something like these to be in every house; feedstock goes in, and food comes out. Remember what I said about post-scarcity? This is a part of the post-scarcity revolution. The part that stops riots over a lack of affordable food.

For the first world, food supply is not a problem. If anything, we're overfed.

For the third world, the problem is mostly getting raw food to people in the first place, and your form of food is not "raw food" but a processed form of food — which they will need a new gadget to use instead of using their native equipment, and these folks' buying power is already jack-shit. There's a very good reason no one tries shipping fully cooked steaks to starving Ethiopians. The problem of food is an environmental problem, a political problem, and a logistical problem. It is not a technical problem, and a new gadget will not solve it.

[Junghalli]

The problem of food is an environmental problem, a political problem, and a logistical problem. It is not a technical problem, and a new gadget will not solve it.

An interlocking complex of new gadgets (dirt-cheap energy, highly autonomous robots, devices capable of producing food from raw elements) might make it a lot easier to solve, but for the moment one basically might as well wish for magic so I don't think we really disagree much here.

[Wyrm]

The problem of food is an environmental problem, a political problem, and a logistical problem. It is not a technical problem, and a new gadget will not solve it.

An interlocking complex of new gadgets (dirt-cheap energy, highly autonomous robots, devices capable of producing food from raw elements) might make it a lot easier to solve, but for the moment one basically might as well wish for magic so I don't think we really disagree much here.

If the capital is too expensive, poor countries will not be able to take advantage of them no matter how cheap the unit cost is. Also, what will they sell to get capital (or commodities!) that if the worth of everything they could possible create nosedives? You can't always depend on philanthropy to pull through.

And again, how do you get it to those countries without political and logistical schenanigans mucking things up, and how do you deal with the nano-castings?

[Ariphaos]

You can learn higher maths this way because you have a teacher who knows his stuff to guide your way. In other words, the problem has already been solved, and your only task is to learn the material.

But the "trial and error" that is described by Alyrium is in regards to improvements on a design that the humans have already busted their balls programming to the best of their understanding. That's a horse of a different color. Apart from a minimal amount of guidance the humans may provide, the computer is all on its own and completely blind, and experience has shown the best way to solve these kinds of problems is trial and error.

Calling the guidance that the sum total of humanity's accumulated knowledge would provide 'minimal' risks trivializing what we've built so far, but yes, there's nothing in particular stopping a rational agent learning from nature rather than humanity, outside of any ethics functions that may be limiting it.

Alyrium's statement is that 'rational AGI is probably impossible for us to realize'. The emphesised statement is important. We may simply not be smart enough to realize a rational AGI, even if it is possible in theory.

Fortunately, it's not necessary for a single human to be smart enough. It may not even be necessary for the sum total of humanity to be smart enough.

First off, we're not really rational ourselves. For the most part, our reasoning is post-hoc if it occurs at all. Only those of us who have trained our fundamentally flakey survival machine called a 'brain' to simulate rational thought can pretend to be rational. Even for them, their prejudices, ambitions, laziness, and so forth can get in the way to foul their reasoning. This is why we have the peer review process in the first place: to catch each other's mistakes.

I'm not sure if there is any place easier to correct for our biases than in computing. Entire departments and sometimes entire businesses are dedicated to testing. A great deal of testing is automated, even.

The second point is that intelligence may not be a step-by-step list of instructions you can write down. There are many other types of algorithms that are not amenable to digital computing. We won't know for sure until intelligence is nailed down and understood.

You can translate any neural network into a set of logic functions. Doing this to a human could require the mass energy of a star, but then if humans are intelligent, it is possible by demonstration.

It hasn't been fruitless to try, either. The search alone has given useful tools.

Self-analysis is extremely limited, if it is applicable at all, due to Rice's theorem. There's no general algorithm to analyze a general algorithm to see if the partial function it implements has any non-trivial property, of which friendliness is one. This lies at the core of computational theory, and so its hard to dismiss with the assertion that "programming techniques improve every year." Programming techniques will not allow you to do the impossible no matter how sophisticated they are. We must first prove that friendliness is amenable to that kind of analysis.

Now you're switching to a different argument. I didn't say much about friendliness at all in my post, though for toy scenarios, friendliness as a function of resource usage is a much easier problem, and even such a 'toy' would be immensely valuable.

In fact you seem to be assuming that attempts to prove friendliness will be at the fully general, unrestricted access to all available resources level. That's insane. Even if it was possible it would be insane. The AI psychoanalyzing a human patient is going to be playing by different rules than the AI managing a Dyson swarm.

The substance directly extruded by these methods are plastic. Do you really expect we will have no need for machined metal in the future?

Of course not. There are plenty of other techniques used in microfabrication. Extrusion is just cheap.

Both websites read exactly like the overly-optimistic tripe that futurists often fall prey to. While home replication will have its place, it cannot replace industry.

Which is fine, because it doesn't need to.

The materials we must work to achieve our high-tech society are too diverse for one small machine to be able to work them all.

Which is fine, because it doesn't need to.

It cannot cure ceramics

Which is fine, because it doesn't need to.

it cannot cut steel rods

Which is fine, because it doesn't need to.

it cannot forge gears

Which is fine, because it doesn't need to.

and it cannot etch ICs.

Nice how you said IC rather than circuit, but again, in your lineup of strawmen - this is fine, because it does not need to.

One machine will never be able to do everything, but that's hardly required or even desirable. In the last discussion on this, it seemed obvious that you would instead have local industries specializing in various small scale gear. It's insane - and a tremendous waste - for every family in a village to be able to craft an engine block, when one or two would be more than enough.

Yes, and look what happened to home computing. It fell by the wayside. Do not dismiss the power of the economy of scale.

I'm not sure what you're referring to. That nearly every person in Western civilization owns a computer is 'wayside'?

[ThomasP]

As far as I've been reading, contemporary thoughts on nanotech manufacturing aren't so reliant on "swarm of nanobots that magic-wands a car" or whatever the nano-wank usually involves. The trend has shifted towards integrated "nanofactories" which would be some kind of 3D printers and successors to the current desktop fabricators like RepRap. All the fabrication elements would be contained within the unit and its controlled conditions, so you wouldn't have to worry about free-range 'bots constructing something from scratch out in the wild environment. It'd be much more organized.

The proponents are also not so interested in duplicating existing industry so much as supplanting it with what Drexler talks about as "diamondoid carbon". So you wouldn't be duplicating steel or ceramics or what have you, but rather creating a whole new series of products based on that class of material.

I'm not speaking to the practicality or feasibility of such things, but I thought it was worth clarifying the position.

[Junghalli]

If the capital is too expensive, poor countries will not be able to take advantage of them no matter how cheap the unit cost is. Also, what will they sell to get capital (or commodities!) that if the worth of everything they could possible create nosedives? You can't always depend on philanthropy to pull through.

It would be more that sufficiently advanced technology could in theory lower the philanthropy effort needed to solve the problem to something plausible for real humans. Our society does try to alleviate famine in the world but it does not have the will to expend enough resources to actually solve the problem, or perhaps it does not have the resources period. If we had vastly more resources then the percentage of our resources that is currently devoted to addressing the problem could be enough to solve it. In the case of food the necessary resources are ultimately the base elements, energy and machinery (in the present world mostly of the organic variety) to turn them into forms we metabolize, and manpower and infrastructure to distribute that. Sufficiently advanced chemical synthesizers and cheap energy could in theory solve 1 and 2, sufficiently cheap robots could in theory solve 3, and a combination of all three could in theory solve 4 by making infrastructure much cheaper to build.

This neglects complicating factors like convincing Kim Jong Mugabe to allow your philanthropic robot army to distribute a year's supply of synthetic nutrient syrup to everyone in his insular fortress shithole.

Of course this is all strictly conjectural (and therefore unproven) and so far from anything presently feasible it's not even worth discussing in the context of presently plausible solutions. So I don't think we're fundamentally disagreeing about anything.

[Wyrm]

Calling the guidance that the sum total of humanity's accumulated knowledge would provide 'minimal' risks trivializing what we've built so far, but yes, there's nothing in particular stopping a rational agent learning from nature rather than humanity, outside of any ethics functions that may be limiting it.

I'd like to point out that, as far as finding the optimum configuration of parts and programming for a good, friendly AI, knowing the general lay of the land (as the sum total of human knowledge would give initially) is indeed 'minimal,' as in its the minimum amount of information you would need to solve the problem. Remember, you're setting this thing to search for better algorithms precisely because you don't know what they are in the first place.

Alyrium's statement is that 'rational AGI is probably impossible for us to realize'. The emphesised statement is important. We may simply not be smart enough to realize a rational AGI, even if it is possible in theory.

Fortunately, it's not necessary for a single human to be smart enough. It may not even be necessary for the sum total of humanity to be smart enough.

I did not mean "us" (underlined) to mean "a single human selected out of humanity;" I mean "humanity in toto." Even collected, we may not be smart enough to crack the problem. Knowing that something is possible and in broad strokes how to do it is a quite different thing from being able to pull it all together.

I'm not sure if there is any place easier to correct for our biases than in computing. Entire departments and sometimes entire businesses are dedicated to testing. A great deal of testing is automated, even.

And yet bugs slip through anyway. Computers are very complex systems, and as the interaction between its parts gets more intricate, mathematical chaos can set in.

You can translate any neural network into a set of logic functions. Doing this to a human could require the mass energy of a star, but then if humans are intelligent, it is possible by demonstration.

A brain is not a neural network as defined in computer science textbooks. It is a physical machine of action potentials floating in a modulating broth of fluid. That's not amenable to translation into logic functions.

It hasn't been fruitless to try, either. The search alone has given useful tools.

Never said that it isn't fruitless.

Now you're switching to a different argument. I didn't say much about friendliness at all in my post, though for toy scenarios, friendliness as a function of resource usage is a much easier problem, and even such a 'toy' would be immensely valuable.

If you test-run your friendliness definition on 'toy' scenarios, how can you be sure that they'll work for real scenarios?

In fact you seem to be assuming that attempts to prove friendliness will be at the fully general, unrestricted access to all available resources level. That's insane. Even if it was possible it would be insane. The AI psychoanalyzing a human patient is going to be playing by different rules than the AI managing a Dyson swarm.

You don't seem to get what I'm saying.

Let's assume for a minute that we can convincingly prove that a particular AI will only produce task-appropriate output. To take your example, a psychoanalytic AI will not generate output to control a Dyson swarm, and vice versa. Thus we have proven that the psychoanalyitic AI will not be able to produce unfriendly Dyson swarm output. But in order to be wholely friendly, the AI cannot produce unfriendly psychoanalytic output either. That's still a property of a partial function that the algorithm implements, and there's no way to decide that the algorithm has this property and not have it be wrong sometimes — and there's no prior guarantee on which algorithms it will decide wrongly on.

Now, it is possible that a proper restriction of algorithms considered and/or inputs fed into the algorithms can let one use an algorithm that need only work for that subset, there's no prior guarantee that the partition of algorithms for which your method decides correctly and decides wrong will fall where you want them to fall. It certainly will not fall along the lines of psychonanalytic AIs/Dyson swarm control AIs.

One machine will never be able to do everything, but that's hardly required or even desirable. In the last discussion on this, it seemed obvious that you would instead have local industries specializing in various small scale gear. It's insane - and a tremendous waste - for every family in a village to be able to craft an engine block, when one or two would be more than enough.

The point is that these rapid prototypers will not break the greedy capitalists' hold on industry and bring it to the masses. It also does nothing to bring about the post-scarcity society you trumpet about. The only reason for a poor country to buy a factory is to generate income. If it just wants a few parts, it can get it cheaper overall from the big manufacturers.

Your rapid prototypers are just that — prototypers. They're only useful for objects that will have very few instances of, basically unique objects. It's good if you have a relatively large number of objects that are unique, exchanging a higher unit cost and capital cost for the prototyper for the ability to make many objects.

For a society to be truly be called post-scarcity, however, the needs of the great majority people must be satisfied, and most people will have the same basic needs. This means that basic necessities in such a world will be commodities, and customization can come trivially through permutation of individual commodities. These commodities can be produced for reduced cost through the economy of scale, and as such if you have a reliable infrastructure to deliver freight, it's much cheaper to get shipments of commoditites than to make them yourself with a rapid prototyper.

And if you have no reliable freight infrastructure, how do you expect to get your rapid prototyper?

I'm not sure what you're referring to. That nearly every person in Western civilization owns a computer is 'wayside'?

No, that nearly everyone who owns a computer owns a mass produced computer. Durable goods are worlds different from software.

=====

If the capital is too expensive, poor countries will not be able to take advantage of them no matter how cheap the unit cost is. Also, what will they sell to get capital (or commodities!) that if the worth of everything they could possible create nosedives? You can't always depend on philanthropy to pull through.

It would be more that sufficiently advanced technology could in theory lower the philanthropy effort needed to solve the problem to something plausible for real humans. Our society does try to alleviate famine in the world but it does not have the will to expend enough resources to actually solve the problem, or perhaps it does not have the resources period. If we had vastly more resources then the percentage of our resources that is currently devoted to addressing the problem could be enough to solve it.

Unless your post-scarcity society is hyperbolically super-science, post-scarcity probably will not obviate the need for resource management. You can have your resources be abundant enough to satisfy your own needs at very little cost to yourself, and still not be in a position to be philanthropic. For instance, if your society achieved post-scarcity not only through automation but also by careful population control and recycling of waste, you can be post-scarcity by any pratical definition and yet phianthropic gifts to other societies may disbalance the resource pool enough to endanger the post-scarcity. Also, resource management by recylcing will mean that the resources cost nothing as long as your resources are not stressed — if you have a protracted period of net iron need such that your iron reserves are depleated, iron is going to cost something nontrivial again.

In the case of food the necessary resources are ultimately the base elements, energy and machinery (in the present world mostly of the organic variety) to turn them into forms we metabolize, and manpower and infrastructure to distribute that. Sufficiently advanced chemical synthesizers and cheap energy could in theory solve 1 and 2, sufficiently cheap robots could in theory solve 3, and a combination of all three could in theory solve 4 by making infrastructure much cheaper to build.

In theory. But any society that you're bequeathing to is not going to be post-scarcity by definition, and has all the problems that come along with it, like corrupt politicians and war. Your bots will have to be guarded, and will have to be inspected and maintained along with your infrastructure.

That infrastructure is also going to be a drain on some non-renewable resources, like metals. If a robot is lost, then you're out its resource investment. If you're achieving post-scarcity by means of recycling, this is capability permanently lost to you. Your convoys will have to be guarded, ect. And because you are guarding these convoys precisely because you don't want material losses, this is going to put a burden on your resource pool.

This neglects complicating factors like convincing Kim Jong Mugabe to allow your philanthropic robot army to distribute a year's supply of synthetic nutrient syrup to everyone in his insular fortress shithole.

Can that nutrient syrup be eaten raw or can be prepared in an ordinary third-world kitchen? Otherwise you're going to need some sort of processing to be able to eat, which requires a gadget. Also, are your tubes recycleable? That implies you need resource management. And if not, where are you getting your materials to make these tubes?

Of course this is all strictly conjectural (and therefore unproven) and so far from anything presently feasible it's not even worth discussing in the context of presently plausible solutions. So I don't think we're fundamentally disagreeing about anything.

Maybe not, but I think there are details that need to be addressed.

[ThomasP]

A brain is not a neural network as defined in computer science textbooks. It is a physical machine of action potentials floating in a modulating broth of fluid. That's not amenable to translation into logic functions.

This is true, but even if modeling the coarse input/output functions of single neurons isn't sufficient (which it may or may not be), what's to stop you from eventually modeling the whole thing at the molecular scale?

I realize that you're talking about substantially more computing power to do that, and that it is in no way trivial due to the complexity of intracellular networks, but if we assume that computing power has enough room to improve (and by all accounts it does) you'd think that sooner or later it would be done.

[Formless]

This is true, but even if modeling the coarse input/output functions of single neurons isn't sufficient (which it may or may not be), what's to stop you from eventually modeling the whole thing at the molecular scale?

The fact that its the molecular fucking scale? Jesus, do you know how many molecules there are in a brain? In a cell? In a nucleus? Good grief. To say that that would take an enormous amount of processing power is not doing it justice. Just in terms of energy, it would be more efficient to just grow a goddamn brain and run it in its natural environment (a body). I mean, a single piece of DNA stretched out would reach the moon, even if you did strip out all the junk instructions (much of which may only be "junk" 90% of the time until its needed in some obscure pathway we haven't yet discovered).

Right now simulating anything at that scale takes a computer the size of a room. That's not going to change any time soon.

[ThomasP]

This is true, but even if modeling the coarse input/output functions of single neurons isn't sufficient (which it may or may not be), what's to stop you from eventually modeling the whole thing at the molecular scale?

The fact that its the molecular fucking scale? Jesus, do you know how many molecules there are in a brain? In a cell? In a nucleus? Good grief. To say that that would take an enormous amount of processing power is not doing it justice. Just in terms of energy, it would be more efficient to just grow a goddamn brain and run it in its natural environment (a body). I mean, a single piece of DNA stretched out would reach the moon, even if you did strip out all the junk instructions (much of which may only be "junk" 90% of the time until its needed in some obscure pathway we haven't yet discovered).

Right now simulating anything at that scale takes a computer the size of a room. That's not going to change any time soon.

That's why the next paragraph, which you conveniently left out, was included, which addressed this.

ETA: In any case, I don't think it would actually come to that, since you could likely model the brain at a much more crude scale, but the point was to note that it could become feasible at some stage in the process.

[Formless]

No, the problem is that you are talking about a scale at which everything becomes so complex that there is no way short of "RAR, majical computors!!!" that this is going to be feasible. End of story.

[ThomasP]

No, the problem is that you are talking about a scale at which everything becomes so complex that there is no way short of "RAR, majical computors!!!" that this is going to be feasible. End of story.

Well that's certainly a compelling argument. Is there any particular reason that the complexity couldn't be modeled by sufficiently advanced processing capability, or do you just not like the idea and assume it's impossible?

Noting, again, that I don't think this is likely to be 1. necessary for brain emulation in the first place or 2. in the range of possibility for a long time if it is, but I'm curious now as to why you're so absolutely certain that you couldn't brute-force the issue with enough processing power.

[Formless]

Are you ever going to demonstrate that by "sufficiently advanced" you don't mean "handwavium magic blackboxes"?

No seriously. For the love of sanity, learn what the words "scale" and "orders of magnitude" mean, then get back to me.

[ThomasP]

Are you ever going to demonstrate that by "sufficiently advanced" you don't mean "handwavium magic blackboxes"?

No seriously. For the love of sanity, learn what the words "scale" and "orders of magnitude" mean, then get back to me.

So you don't actually have any evidence that it's computationally impossible with enough processing power thrown at the problem?

[Wyrm]

A brain is not a neural network as defined in computer science textbooks. It is a physical machine of action potentials floating in a modulating broth of fluid. That's not amenable to translation into logic functions.

This is true, but even if modeling the coarse input/output functions of single neurons isn't sufficient (which it may or may not be), what's to stop you from eventually modeling the whole thing at the molecular scale?

I realize that you're talking about substantially more computing power to do that, and that it is in no way trivial due to the complexity of intracellular networks, but if we assume that computing power has enough room to improve (and by all accounts it does) you'd think that sooner or later it would be done.

No. Even if you could abstract the functioning of a single neuron to a comparatively short algorithm, there are many combinations of connections to consider, so the complexity explodes at O(n!) — very, very fast, such that the possible connections for for a 100 billion neurons is one huge ass number: ~10^(9e10). This is waaaaaay larger than the number of plank times the universe has existed times the number of particles in the observable universe (<10^200). The actual combinatorics are quite a bit less, but they're still huge compared to the theoretical computational power of the universe. Even if there were only 100 neurons to connect up, there are still over 9.3326215443944e157 possible permutations of them.

[Junghalli]

Here's a link to that video on brain emulation I mentioned a while back.

Unless your post-scarcity society is hyperbolically super-science, post-scarcity probably will not obviate the need for resource management. You can have your resources be abundant enough to satisfy your own needs at very little cost to yourself, and still not be in a position to be philanthropic.

Such scenarios are indeed possible, but that has little to do with the point I was making. I was pointing out that in theory you could postulate a scenario where technology would solve world hunger. It would require a large number of speculative and therefore unproven technologies, but it is imaginable and doesn't, as far as we can tell, require you to break physics. I said nothing about the plausibility of such a scenario, I was simply pointing out that it's theoretically possible without resorting to blatant magic of the sort you need to justify something like, say, FTL.

That infrastructure is also going to be a drain on some non-renewable resources, like metals.

OK, add another term to the requirements: the technology to economically obtain useful elements from trace concentrations or access to the resources of the greater solar system. Either one in combination with the others could theoretically make this a trivial issue.

Can that nutrient syrup be eaten raw or can be prepared in an ordinary third-world kitchen? Otherwise you're going to need some sort of processing to be able to eat, which requires a gadget. Also, are your tubes recycleable? That implies you need resource management. And if not, where are you getting your materials to make these tubes?

The exact answer is totally irrelevant except insofar as no, it wouldn't be something that would require extensive preparation (I might as well have said synthesized gourmet dinners as far as the point of the scenario is considered), but when I said that I was thinking of something along the lines of this.

[ThomasP]

A brain is not a neural network as defined in computer science textbooks. It is a physical machine of action potentials floating in a modulating broth of fluid. That's not amenable to translation into logic functions.

This is true, but even if modeling the coarse input/output functions of single neurons isn't sufficient (which it may or may not be), what's to stop you from eventually modeling the whole thing at the molecular scale?

I realize that you're talking about substantially more computing power to do that, and that it is in no way trivial due to the complexity of intracellular networks, but if we assume that computing power has enough room to improve (and by all accounts it does) you'd think that sooner or later it would be done.

No. Even if you could abstract the functioning of a single neuron to a comparatively short algorithm, there are many combinations of connections to consider, so the complexity explodes at O(n!) — very, very fast, such that the possible connections for for a 100 billion neurons is one huge ass number: ~10^(9e10). This is waaaaaay larger than the number of plank times the universe has existed times the number of particles in the observable universe (<10^200). The actual combinatorics are quite a bit less, but they're still huge compared to the theoretical computational power of the universe. Even if there were only 100 neurons to connect up, there are still over 9.3326215443944e157 possible permutations of them.

Cool. That makes sense.

Could that conceivably change if you brought in some hefty quantum computers and associated algorithms? It sounds like the kind of thing they'd be good at.

[Wyrm]

Here's a link to that video on brain emulation I mentioned a while back.

Unless your post-scarcity society is hyperbolically super-science, post-scarcity probably will not obviate the need for resource management. You can have your resources be abundant enough to satisfy your own needs at very little cost to yourself, and still not be in a position to be philanthropic.

Such scenarios are indeed possible, but that has little to do with the point I was making. I was pointing out that in theory you could postulate a scenario where technology would solve world hunger. It would require a large number of speculative and therefore unproven technologies, but it is imaginable and doesn't, as far as we can tell, require you to break physics. I said nothing about the plausibility of such a scenario, I was simply pointing out that it's theoretically possible without resorting to blatant magic of the sort you need to justify something like, say, FTL.

Yes, technology may (or may not) make your task easier, but they cannot solve fundamental social problems.

That infrastructure is also going to be a drain on some non-renewable resources, like metals.

OK, add another term to the requirements: the technology to economically obtain useful elements from trace concentrations or access to the resources of the greater solar system. Either one in combination with the others could theoretically make this a trivial issue.

Why?

The exact answer is totally irrelevant except insofar as no, it wouldn't be something that would require extensive preparation (I might as well have said synthesized gourmet dinners as far as the point of the scenario is considered), but when I said that I was thinking of something along the lines of this.

Looks boring. I hope they have flavor of the month.

[Wyrm]

Cool. That makes sense.

I have to throw in the caveat that my figures are probably a horridly bad highball. I admit to frustration as my computer choked on the figure, even with BigNums. I'll probably come up with a better ballpark tomorrow, but be warned that it's not going to be pretty either — Combinatorics BAD!

Could that conceivably change if you brought in some hefty quantum computers and associated algorithms? It sounds like the kind of thing they'd be good at.

Quantum computers have their own problems, such as needing to give up determinism. There are also decoherence issues; you need a very delicate setup in order to perform quantum computing, which includes no excess heat. Also, if we're simulating whole neurons, we're going to need a timeslice on several switches for each and every particle in the neuron, as well as storage space to store that particle's particulars.

This is quite doable for a system of a couple hundred to a thousand atoms, as long as you're talking about short timescales, because we know a lot about how atoms behave and only a few possibilities for each bond.

(Don't quote me on this, but I remember reading somewhere that to simulate the universe requires a universe-sized computer, and remember that the universe is mostly empty space!)

[Junghalli]

Why?

Because the resources of the greater solar system should be quite generous compared to the needs of a planetary scale civilization like ours. Unfortunately the Asteroid Mining For Profit page with its nice chart seems to be down, but consider as an example the asteroid 16 Psyche, which seems to be essentially a flying 200 km mountain of nearly pure metal, with an estimated mass of 2.19 X 10^16 tons; equivalent to millions of years of present day annual iron extraction even if only a fraction of it is useable.

Looks boring. I hope they have flavor of the month.

Beats starvation though.

[Formless]

So you don't actually have any evidence that it's computationally impossible with enough processing power thrown at the problem?

Wyrm nailed it before I could (I was eating dinner). I'm not saying that its computationally impossible in theory given arbitrarily large amounts of processing power and energy. I'm saying that "arbitrary processing power" is no-limits wank, and therefor has no place in a serious discussion.