StarDestroyer.Net BBS

Does anyone have experience with building a cluster setup?

The sort of thing I'm looking to do is to be able to have a bunch of independent nodes that can each handle independent single threaded jobs. I've used clusters like these in the past administered using Condor to manage jobs running on nodes.

To start off with I was thinking about getting some hardware and setting up a mini test version with maybe 3 nodes and having a look at how it would likely perform and be to administrate. If that works out then I can probably spend much more on a full scale version. My technical requirements for the sorts of things I'm looking to use it for are just that it can support ~2GB ram on each node and a reasonable (but probably never more than 100GB) amount of HD space as well. Processor speed is always welcome but doesn't have to be great, and I don't really need more than single core processors on each node. GPU is also unnecessary.

Does anyone know what a good source of the individual machines would be? I'm looking to try and get something for £50-100 a node (ignoring associated housing, power and networking costs).

Looking around it would be possible to get a self contained node as cheaply as £20 if it were something like a Raspberry Pi, (although the capabilities of that are too limited for my purposes).

The pandaboards from this build are the sort of thing I'm looking for.

Does just buying an ITX board give you all the stuff to make one complete node ready to hook into place?

Is the alternative of bulk buying a bunch of low end desktop components and just putting them in a small case going to get comparable results for a non specialist application?

Also from the perspective of linking it all up, what sort of networking hardware is best?

Anyone who has any experience with this sort of thing would be really helpful.

You might want to research if which of the two different architectures you listed ARM or x86_64 is better at doing the computing you are looking at. From the point of view of expandability, x86 is probably the way to go. ARM stuff rarely has the ability to change core system components (memory, network cards, or being able to directly attach hard drives)

On the x86 side of things, you are probably better off going with an intel atom motherboard over fusion. The advantage fusion has is the integrated GPU which you said you don't need. Intel makes some quite cheap motherboards. Slap some ram in there, give it power, enable network booting (Or add hard drive, though using netboot and nfs might make it easier to expand later on), and you'd be all set. Otherwise, you might get better results with the cheapest discrete Intel processors and motherboard you can find.

I'd suggest looking at either the Guruplug or the beagleboard-xm but those both limited by memory.

Save yourself some headaches and get a decent quad core setup. One machine is much much simpler to administrate than 2-3. From your brief description, it sounds like all your looking to do is some simple batch processing with 2-3 batches running at once. A top end core i5 based system with 8GB RAM can be had for about $700 US if you use the onboard graphics. If you really want the separate machine route, its quite possible to do Atom systems for around $200 US and still hit your RAM targets. All prices assume no Windows.
For your interconnect question, since it appears none of the nodes will need to talk to each other, other than to distribute batch jobs, gigabit or even 10/100 are fine. This is looking like you want a cluster for a cluster's sake and not because it is suited to your problem set.

First you should closely examine your problem to make sure clustering is the best solution for your problem. I think your next step should be determining what middleware you will use, or the software that manages and holds the cluster together, rather than choosing the hardware. Some middlewares have hardware requirements that each node must meet. If you have a decent computer you can set up several virtual machines to test your cluster and some middlewares support single node clusters for testing purposes.

On the hardware side I don't have experience on choosing hardware for clusters. The cluster I work with has 4 nodes and 32GB of RAM per node, so I don't know the performance of low end node clusters, but I assume that it would be dependent on the middleware and how many nodes resources are split across.

Going back to the problem you want to solve (or types of jobs you want to run), I would suggest you get a higher end computer and run as many threads as you have cores. Write a manager program or script that starts new jobs as older jobs complete and manage errors. I think clustering is more applicable when a problem can be parallelized, or be split into subset that can run independently and be recombined for a solution.

If you're looking for a cluster middleware to play with, I've used Apache's Hadoop which is written in Java and uses the MapReduce paradigm. Programming for Hadoop is an exercise in defining your problem in map and reduce phases. Hadoop, as the middleware, manages the starting of map and reduce jobs and communication between nodes.

Steel wrote:Is the alternative of bulk buying a bunch of low end desktop components and just putting them in a small case going to get comparable results for a non specialist application?

No. A non-specialist application is almost certainly going to perform best on CPUs with good serial performance (big, brawny ones, not wimpy ones like ARM or Atom).

Performance of a cluster of such low-end hardware will almost certainly be worse than the equivalent priced comodity PC or servers, probably much worse, and obviously it will be much more effort to build, configure and administer. This is why I use individual machines with 6 or 8 top-end GPUs for compute tasks instead of a couple of hundred ARM crapboxes. Several well-funded startups have tried to make servers with a few hundred ARM or Atom cores with better price/performance than any home-built equivalent, but they still aren't getting much takeup because the massive duplication and communications overhead is inherently inefficient. Supercomputers are only made out of clusters because they're already at the limits of individual node performance. If your intent is to test software for a future cluster of full-power nodes, I suggest using a set of virtual machines on a single conventional server, or renting temporary space on a cloud compute service.

Starglider wrote:Performance of a cluster of such low-end hardware will almost certainly be worse than the equivalent priced comodity PC or servers, probably much worse, and obviously it will be much more effort to build, configure and administer. This is why I use individual machines with 6 or 8 top-end GPUs for compute tasks instead of a couple of hundred ARM crapboxes. Several well-funded startups have tried to make servers with a few hundred ARM or Atom cores with better price/performance than any home-built equivalent, but they still aren't getting much takeup because the massive duplication and communications overhead is inherently inefficient. Supercomputers are only made out of clusters because they're already at the limits of individual node performance. If your intent is to test software for a future cluster of full-power nodes, I suggest using a set of virtual machines on a single conventional server, or renting temporary space on a cloud compute service.

I think this is a little misleading. You don't need to be in the supercomputer range to benefit from a cluster. But to benefit the most from a cluster, you should use higher end machines. For example, it would be rather expensive to build a single system with 128 GB of ram and 64 cores, but having 4 machines with 8 cores and 32GB each would much cheaper. You can't scale too far downwards though, if a single machine with the ram and core count can be built for less than the cluster, go for the single machine.

TronPaul wrote:I think this is a little misleading. You don't need to be in the supercomputer range to benefit from a cluster. But to benefit the most from a cluster, you should use higher end machines. For example, it would be rather expensive to build a single system with 128 GB of ram and 64 cores

That spec is actually at the top end of the commodity server bracket; a quad-socket AMD Interlagos server fully populated (all 16 channels, those boards usually have 32 physical slots) with 8 GB DIMMs. This is roughly comparable in price/performance to the ubiquitous dual-Xeon rackmount 1U servers that fill about 80% of the data center market; a little better for VM use, a little worse for the typical scale-out web services dross. You start paying a premium above that spec due to a combination of price segmentation and lack of economies of scale.

but having 4 machines with 8 cores and 32GB each would much cheaper.

Not that much. For example specing on the Broadberry site, cost of a 64-core 2.2 GHz Opteron server with 128GB of ECC RAM : £5081.43. Cost of each 8-core 2.7 GHz Opteron server with 8 GB of ECC RAM : £791.55 (with 1 HD and minimal spec otherwise). Cost of a gigabit switch to connect the later together : £50 ish. The former has somewhat less per-thread performance but vastly better inter-thread latency and bandwidth, and more usable memory due to the lack of duplication. However if you did the same thing with Xeons quad-socket wouldn't make sense, because Intel prefers to slap a higher price premium on the EX versions. Obviously you can save more money by using consumer grade hardware instead of server grade hardware - I do with GPUs - but that's no longer a straightforward comparison.

Thanks for all the input guys.

The jobs I have to do are all independent pieces of work, so there is basically no inter node communication, only communication between nodes and the master. There is quite a lot of data to be sent and retrieved from the nodes, but not all at once, so there shouldn't be too much of a bottleneck in the networking. This isn't really for a massively parallelised computation, more for high throughput. For controlling the setup I'd almost certainly use Condor as I have used that in the past and it distributes jobs to the nodes and seems to keep everything running smoothly.

Obviously if I just wanted a few nodes then it would be far easier to get a single high spec quad core machine, the 3 nodes suggestion was just so I could set up a test example and benchmark individual node performance and try out administering the setup before expanding to the full size. My target performance was to see if it is feasible to get the equivalent of 10-20 pcs for a couple of thousand GBP.

I found that you can get a £20 computer with a dinky processor and an apparently decent (although for my purposes unnecessary) GPU in the raspberry pi, and you can get a dual core, 4GB ram, some graphics and HDD in a desktop from DELL unit for £250 (including windows), so surely there must be a way to get some reasonably capable bunch of hardware so that you get significantly more processing power per £ than just buying a few complete desktop PCs?

The Guruplug and beagleboards look like they have too little ram per node to be of much use to me. For the mini-ITX you suggest rapidsquirrel, do those boards at those prices actually come with a CPU already installed?

The Intel D525MW board? Yes, it's got a dual core atom processor soldered to the board.

Though if you are looking for processing power per pound, you are better off as other suggested with processors other then the atom. Most likely, a mid-grade Intel processor would be the best from a price versus performance standpoint.

Though if this is more of a cheap technical demonstration (I'm actually jealous. I've been wanting to build a cluster for a while, but have never had a good reason to), the atoms will be the cheapest way to go about it. You can always add faster units later.

rapidsquirrel wrote:The Intel D525MW board? Yes, it's got a dual core atom processor soldered to the board.

Though if you are looking for processing power per pound, you are better off as other suggested with processors other then the atom. Most likely, a mid-grade Intel processor would be the best from a price versus performance standpoint.

Though if this is more of a cheap technical demonstration (I'm actually jealous. I've been wanting to build a cluster for a while, but have never had a good reason to), the atoms will be the cheapest way to go about it. You can always add faster units later.

Good point that, given the software and the intended use, adding or removing nodes is very easy, so I could just go for something small and easy to begin with. I would like to be able to benchmark something in the range that I was actually looking to get in the end before expanding. I'll shop around a bit for the next few days and see what I can dig up.