Recommendations for a MASSIVE Database

[Solauren]

Hello.

I've got a side-project (more of a 'can it be done' curosity) that I'm trying to do.

However, I found out MS-ACCESS can't handle it. The resulting tables are going to be way, way to big for it to handle. (Several Gigabytes a piece).

Does anyone know of a system that can handle larger files? How well does Visual Basic handle Database functions using a Text back end?

[phongn]

Use PostgreSQL. Don't use MySQL.

[Solauren]

Could it handle a Index List of 7,850,000,000 or so entries?

[phongn]

Could it handle a Index List of 7,850,000,000 or so entries?

That might be more hardware dependent - but Postgres is used for really, really big databases in the Real World. I doubt you'll have trouble with its scalability.

[Starglider]

Use PostgreSQL. Don't use MySQL.

I second that.

Could it handle a Index List of 7,850,000,000 or so entries?

Yes.

What exactly are you trying to do in this project? The words 'Visual Basic' do not go well with 'large data set processing', but if run-time isn't an issue, then it might not matter.

[Solauren]

I've been toying with the idea of making a Text based Turn Based Startergy Game, similiar to Civilizations. However, unlike in Civilizations, you'd be able to design custom Starships for your civilization to use. Think Civilizations meets FASA Star Trek.

It's set in space, with randomly generated Star Sytems.
I was hoping to find a database with the capacity needed to actually hold a true galactic stellar population record.

And yes, I can get pretty damn ambitious.

[Starglider]

I was hoping to find a database with the capacity needed to actually hold a true galactic stellar population record.

Masters of Orion 3 went down this path and it didn't work very well. Either there is a massive amount of data that the player has to trudge through, making the game horribly boring, or everything is automated and the player gets the impression that the game is mostly playing itself. Relying on large scale simulations to produce interesting 'emergent behaviour' is also a bad idea; complicate it enough and you will get unpredictable behaviour, but 9 times out of 10 that behaviour will make the game more confusing and less fun, rather than delivering any improvement in playability.

That's on top of the difficulties of making your algorithms run in a sensible amount of time. Squeezing all that data through the relatively narrow straw of the database's API to do processing on every turn is highly suboptimal for this.

[Ariphaos]

MySQL is limited to four billion rows, though I don't know if that applies per partition - if so you could use four trillion in MySQL 5.1.

Postgres does not have such a limit (outside of seriously impacting performance, or so I'm told), but sometimes you will see references to object ids, which are 32-bit. Don't use them if you are seriously planning on > 4gb rows in a single table.

[Solauren]

Snip

I wasn't planning to include computerized civilizations. Just Player vs Player.

However, it's begining to sound like avialable technology would make this idea, problematic, at best.

At least on the scale I wanted to. I like the idea of 1000 players working from just reaching FTL, fighting each other ala Civilizations, and eventually only 1 player coming out on top.

Sure, it could take a few years, but....

Maybe I need to scale this down..

[Starglider]

However, it's begining to sound like avialable technology would make this idea, problematic, at best.

How much experience do you have in large scale dataset processing and strategy game design? What you're talking about would be extremely difficult for a skilled team of developers with plenty of dev time to pull off. If you have minimal experience and just a few hours a week of your spare time, then the chances of success are almost exactly zero.

[Solauren]

However, it's begining to sound like avialable technology would make this idea, problematic, at best.

How much experience do you have in large scale dataset processing and strategy game design? What you're talking about would be extremely difficult for a skilled team of developers with plenty of dev time to pull off. If you have minimal experience and just a few hours a week of your spare time, then the chances of success are almost exactly zero.

Large Scale databases - modest amount. I work with near-million record databases at work. (3/4 of a million records or more).

Stratergy Game design - next to none.

[Starglider]

Large Scale databases - modest amount. I work with near-million record databases at work. (3/4 of a million records or more).
Stratergy Game design - next to none.

In that case you should decide what you want more, the technical/intellectual challenge of working on a ridiculously ambitious design that you will never complete, or the pleasure of seeing other people have fun playing your (much more modest) game. That's a serious question, there's nothing wrong with starting a project you know you can't complete just for the experience of trying, as long as you're honest with yourself about it.

[Solauren]

Oh I'm well aware of that.

I'm thinking what would probably be best is to ignore the galactic scale for now, and make a ship design / tech tree, and keep things local (say a Sector wide war, that's still fairly big).

If that were to 'take off' so to speak, scaling it up to galactic size might be more feasible.

This is, of course, assuming i don't lose interest in the idea in a week or two. It's been known to happen

At least I now know about the better database software avialable.

[petesampras]

I have to agree with Starglider on this one. Even if you had a team of the best games developers in the world and millions to spend on this game, it would still be shit. MOO style games are fun when you can take a hands on approach and get involved with tactics, science, fleet design, etc. With a galactic sized empire this is just impossible. Every turn you are going to have hundreds of thousands of planets to manage. In a war you will have hundreds of battles every turn to resolve. It makes no sense to get your hands dirty in micro managing any of this. Even if you did for a few planets/battles/ship designs/etc, that meddling would have minimal effect on the overall outcome. The game must inherently play itself, with you making only very global decisions. MOO2 was hopeless unrealistic, with a tiny cramped 'galaxy'. It was also a fantastic game. Ambitious MOO3 - well let's not mention that....

[PeZook]

Holy shit...how come I didn't know about PostgreSQL? How could I miss it? Why did I stick to the miserable failure that is MySQL?!

Jesus christ, I am a retard...

[Sriad]

A still ambitious but much more reasonable setup might be a small globular cluster of, say, 1000 star systems/interesting stellar bodies. You could probably massage tactics in the game so that combat is decided in a small number of major battles instead of hundreds of small skirmishes to allow deeper player interaction.

An in-universe explaination might be that at the tech-level where major conflicts start up, empires develop wormhole networks so massed defenses can quickly respond to attack. Conversely, the goal of an attacking force is to overcome hardened wormhole defenses before a defensive fleet can be mobilized and pumped into the system. Maybe while protecting the wormhole mouth they're being supplied and reinforced through themselves...

The point is, you can do a large number of star systems as long as there are fun mechanics in place such that players only need to make a manageable number of decisions per turn, but decisions with deep tactical value. There is probably another way it could be explained if you had a billion-star empire fighting another billion-star empire instead of 100 vs. 100. Possibly control of galactic artifacts or hyperspace routes; you know your universe better than I.