StarDestroyer.Net BBS

I'm running out of space on my main computer, and was thinking about building a home file server. Throwing together some cheap hardware on newegg, it looks like the total will come to about $150 to $200 depending on how much ram I want and how many cores I want my processor to have. Right now I have 2 500gb hard drives that each have about 100gb empty, plus my os disk. Most of my largest files are things I've downloaded off of Bittorrent and still have the .torrent files for. It would probably be a lot cheaper to just keep multiple copies of the .torrent files and keep the actual files on a single hdd without any mirroring or parity. (or a jbod or just split across multiple hdds based on the letter it starts with or something) I'd like to just keep this box in a closet and access it via remote desktop/vnc/etc when I need to access it at all, mostly I'd just access to network-mounted drives. The biggest things I want recommendations on are:

• Should I use raid, and what type? Raid-z or Raid-5? How many drives does Raid-z require? Do they all have to be the same size? Can I add more drives later? I've been googling raid-z all morning, but can't seem to find the answers to any of these questions, at least not recent ones.
• What hardware do I need? How much ram and how many cores?
• Should I replace my 500gb drives with larger ones?
• What operating system would be best for these purposes?

Don't hesitate to ask if you need any further information information.

- Don't go for RAID unless you're willing to pay extra money just to avoid the hassle of backups* and/or redownloading files.

- If all you're doing is file serving for yourself, you don't need much at all - a Pentium 3 would suffice for that.

- Why replace when you can add? 500GB drives are still relatively big among hard drives... just buy a 1TB drive and add it in. When that gets full, buy and add another one.




*Yes yes yes, RAID is not a backup. But it is a pleasant half-ass middleground for data that isn't really worth the cost of backing up and would be cumbersome to redownload.

RAID-Z would seem to be Sun's answer to RAID-5. So unless you plan on using Solaris, I would guess you don't want that. But then again maybe you are. Since it's equal to RAID-5, that would mean 3 drives minimum. Honestly, unless you are really worried about losing data or attempting to get extra speed, I wouldn't worry about RAID-anything. JBOD would probably work just as well.

The only hardware constraint the I run into is the speed of the network card and ATA type. You don't need anything beyond a p3 to run a file server, but you most likely won't find a p3 with a gigabit network card and you won't find one with SATA. I would recommend a cheap modern AMD system. Just makes sure the number of cores you have is equal to or greater then 1. RAM could easily be in the 512 range, but more is always better.

If you are looking to significantly increase your storage space, you might as well replace your 500GB. Hard drives are getting cheaper by the day. But if you are looking to keeping to a budget, you could easily add another drive later.

Linux, but I'm biased. I'm going to assume that you don't have linux experience(You probably wouldn't have asked the question if you had). If you comfortable with teaching yourself computer stuff and don't mind a few complications during the install process, try Ubuntu and see how you like it. If you would rather not have hassle of learning a new OS just yet, stick with what you are familiar with.

Ultra-cheap components have a considerably higher failure rate (PSUs in particular). Factor in the potential hassle of having to fix them if they break.

Do you really need a full server? Have you considered a NAS box? That comes completely preconfigured and ready to go.

I suggest RAID1, with terabyte drives. Once you've got your stuff copied to that you can format your existing drives as a second RAID1 array, for x3 the space in total. It should suffice for your needs and it's relatively easy to set up in Linux without dedicated hardware; by contrast RAID5 is a bit of a pain.

RAID-Z(2) pretty much requires you run Solaris or FreeBSD. It's a fine solution, but it also is fairly more CPU and RAM intensive (and you need to be using an x64 processor).

Starglider wrote:Ultra-cheap components have a considerably higher failure rate (PSUs in particular). Factor in the potential hassle of having to fix them if they break.

Do you really need a full server? Have you considered a NAS box? That comes completely preconfigured and ready to go.

That could definitely work. Looking on Newegg, the one I like best is this one. DHCP, FTP, Print server, Bittorrent, 2 usb ports, and pretty cheap.

phongn wrote:RAID-Z(2) pretty much requires you run Solaris or FreeBSD. It's a fine solution, but it also is fairly more CPU and RAM intensive (and you need to be using an x64 processor).

That's fine. All I'd need to do is set up raid-z, mount the drive on the network, and install vnc or something. I'm sure there are guides online for how to do all of those things.

Destructionator XIII wrote:Potentially stupid question: why not just add a new hard drive to your existing computer?

No more empty 3.5 inch bays. I could replace one, but would be getting less net space then I'm paying for. Also I wouldn't mind something that had more protection against failure then what I have now. Right now I have no backups or anything.

At my house I use a Buffalo Linkstation NAS appliance. It has a single 500 GB SATA HD and supports gigabit ethernet, and it runs Linux (and of course I've hacked it, and can access it via ssh, run my own scripts, install packages, whatever). The unit replaced my IBM NetVista storage server, which is a Pentium III system with 256 mb of RAM (that unit I still have, but I use it as a console server for my Cisco lab).

The Linkstation is nice because its extremely energy-efficient. It + my five WRT54GL units and my WRT54Gv8 router, plus one or two switches, and the cable modem, are the only devices at my house that run 24/7.

For backups and additional protection, I'm in the process of setting up an NSLU2 NAS appliance I own. The NSLU2 isn't nearly as good, its a much slower system in a tiny little plastic box, and you connect an exteranl USB HD to it (and it only has 10/100 networking), however, it is ideal for use both as a management system for backups and as an emergency failover NAS. That unit will be connected to the Linkstation via a Gigabit switch (my LAN is a WDS so the maximum throughput between systems not connected to the same switch is basically standard wireless G transfer rates, i.e. 10-50 mbps). The NSLU2 will use rsynch to get periodically updated snapshots of data on the Linkstation, so I'll be able to easily restore any accidentally deleted files, and additionally I will archive and compress important files, and scp them to my production environment in Los Angeles. This setup is probably overkill for your purposes.

For a brief period of time, I did the above using my Linkstation NAS and my NetVista storage server, but I needed the NetVista for my Cisco lab, and replacing it with the NSLU2 will result in power consumption savings.

In terms of pricing, by the way, a single drive Linkstation with 500 GB will cost less than $200, and you can also get a unit with dual drives (a major Internet firm has a box for one of these sitting in their cage at the DC in LA where I'm colocated). The dual drive unit can of course do RAID. The NSLU2 is no longer made, but you can get a used or new one for around $50.

Uraniun235 wrote:- Don't go for RAID unless you're willing to pay extra money just to avoid the hassle of backups* and/or redownloading files.

- If all you're doing is file serving for yourself, you don't need much at all - a Pentium 3 would suffice for that.

- Why replace when you can add? 500GB drives are still relatively big among hard drives... just buy a 1TB drive and add it in. When that gets full, buy and add another one.




*Yes yes yes, RAID is not a backup. But it is a pleasant half-ass middleground for data that isn't really worth the cost of backing up and would be cumbersome to redownload.

RAID anything sucks for home use, even RAID1. You're better off leaving 1 drive unmounted and periodically syncing the contents over w/ a script, which gives you some fallback when you accidently delete everything on said drive,suffer filesystem corruption, or other catastrophe other than drive failure.

This from the guy with a 9TB RAID array?

Uraniun235 wrote:This from the guy with a 9TB RAID array?

I've never had RAID array of any kind.

How did I mix you up with Xon?


That said your solution doesn't work so well if he has more files than can fit on a single unmounted drive.

Pu-239 wrote:RAID anything sucks for home use, even RAID1.

Why?

You're better off leaving 1 drive unmounted and periodically syncing the contents over w/ a script, which gives you some fallback when you accidently delete everything on said drive,suffer filesystem corruption, or other catastrophe other than drive failure.

I just use a journalling file system. The probability of ext3 dying without a physical drive failure is very very low. But the setup you mentioned is ok, if you only have two drives, don't mind losing a few hours work/email if and when the primary drive fails, and don't need a x2 read speed improvement. It does have the advantage of using slightly less power, if your OS can spin down the backup drive when not actually doing the backup.

Uraniun235 wrote:That said your solution doesn't work so well if he has more files than can fit on a single unmounted drive.

It is hellish trying to locate a given file over a half a dozen hard drives even with good sorting. Been there done that, and wrote a fucking script to generate a single Windows DFS tree linking ~6 drives into the one file tree automatically.

Xon wrote:It is hellish trying to locate a given file over a half a dozen hard drives even with good sorting. Been there done that, and wrote a fucking script to generate a single Windows DFS tree linking ~6 drives into the one file tree automatically.

Windows Search is pretty good nowadays about searching across things.

When I was talking about RAID sucking, it was in the data backup context and how syncing periodically is superior. Obviously it's better to RAID if you want the read/write performance, although most users won't need that (often bottlenecked by 10/100 ethernet anyway (in the context of home servers)).

If you have multiple drives that's different, and using schemes other than RAID0 or JBOD to combine drives wastes more than 1/2 of the total drive space if used w/ the above syncing scheme for backup (which still sucks if you have a PSU fry all the drives in the system or fire/water damage or other large catastrophe).

Switchfs seems like a better alternative to JBOD, since it seems to retain data on the drives that haven't died since files remain on physically seperate filesystem.
http://code.google.com/p/sfs/ . Doesn't seem mature though so I wouldn't use it. There's also UnionFS if you just want to read off the concatenated partitions.

I'm a huge fan of RAID5 but that's probably due to the fact that I work with it in a professional environment. Being a sysadmin instills a fair amount of paranoia about the reliability of individual disks and most of my home data I don't consider important to waste tape space on.