How We Built A Data Center With Commodity Hardware And FOSS

Interesting write-up, but I expect you will have some problems over time with this.

In my experience the computer failures are relatively easy to manage. Power will be your primary issue, and to a lesser degree, internet. When we ran a lot of servers on UPSs, the UPSs start to become a major pain. UPSs fail much more often than hard-drives. And when the battery dies it cuts off power to the computer. You're using low-end desktop machines, so you probably don't have dual PSUs in each machine (if you did you could run each PSU to a different UPS). UPSs are also much more likely to fail when the grid power fails. So the power goes out, your UPSs will kick in, then 30 seconds later one or TWO fail. Can your power distribution can handle that? I'm no UPS expert, but this seemed pretty common across a variety of UPS products.

As for internet, multiple links is a good start. Load-balancing outbound traffic is fairly straight-forward. Inbound traffic is much harder. Really you need BGP. A BGP-based set of links isn't cheap, and doing the configuration yourself is hard if you don't have experience with it. The alternatives that aren't IP routing based are usually based on DNS, which is better than nothing. But the effectiveness of it is dependent on how your users servers/clients/browsers implement DNS caching. It will work OK for some users but some will cache the stale link IP for 24 hours -- not cool.

You certainly can work this way, and probably get three nines of uptime. But I can say now that we switched to a decent colo I sleep a lot better. We can focus more on building product, adding value for our customers, than screwing around with the utilities.

FWIW I'm basing this off of my experience in the United States.

The very first thing you should do if you want to bolt-together your own hardware out of commodity devices is invest in static control technology. This is often not understood by those without experience in electronics manufacturing. Static electricity can cause damage to components that will not surface for months. Because of the nature of the mechanism it is almost impossible to know that static was the culprit when a motherboard, memory or drive fail.

In our case, we, among other things, do a lot of hardware manufacturing (as in buy the chips and build our own boards). Because of this we invested in custom conductive flooring and everything else above the floor is also static-aware. For example, all rolling carts and shelves are wired with redundant contacts that touch the floor, thereby making a connection.

The above is an extreme for the occasional builder. At a minimum you should setup a working environment that has a conductive surface like an static-dissipative mat and wear a conductive wrist strap connected to the mat as well. There are also ionizers and other tools that would be good investments.

Handling components around server racks should never happen without regards for static electricity safety. The rack should be grounded and you should wear a conductive wrist strap connected to the rack.

I'll bet that a lot of failures in consumer/commodity builds are due to this very issue and not necessarily connected to bad or questionable components. If you buy components built by reputable manufacturers they will have been manufactured in tightly controlled environments already. Don't break the reliability chain by handling these components in uncontrolled environments.

The problems you will face are not obvious and immediate problems. For example, a year from now, a server might fail, and you find out that you can no longer obtain that specific model of motherboard or RAID card that was used in that server, so you have to replace the entire thing (purchase all new). Or, you find that a drive in your RAID array fails and you can no longer get a drive with the exact same model and disk geometry as the other disks in the RAID, so you are unable to rebuild the RAID and have to basically start from scratch.

The main reason why you should go with real server type components is the support lifetime. You can be fairly confident that you can get replacement parts be it CPU, memory, raid controller, motherboard, or disk drive, for at least 5 years after purchase. Otherwise you will find that when your components die, you will be completely unable to repair them due to lack of available parts.

Good luck to you, but I've been down this road before and if you cut corners in the beginning, it usually costs you more money to do it right later on.

My questions about the data center are what kind-of climate controls they're using (not listed in the article) and how they're making sure that the generator and UPS are always in good condition. At my previous job we were able to correlate many component failures to poor climate conditions (too hot, too cold, too little humidity, too much humidity).

I also recall back in college setting up a 110 node linux cluster. Before they upgraded the HVAC system in the server room the cluster generated so much heat you the power cables started to shows signs of heat damage (we had to shut the cluster down until the HVAC was beefed up).

I've also had a bunch UPS where the batteries fail, and need to be replaced periodically. I also wonder how that figures into the cost of their data center (it's an issue at a big data center as well).

Looks a bit shonky to me, but if it works, good for you.

I worked for a design agency back in about 2004 which had the same sort of infrastructure internally. They had a cheap desktop Exchange 2003 box, cheap desktop Fileserver, cheap desktop SQL + App server, cheap desktop mp3 server, big old DLT drive, SDSL router, switch, a KVM, cranky old monitor, keyboard, mouse, mounds of cable, a well underpowered UPS and a garfield all wedged under a couple of tables.

Some muppet knocked over a coffee cup and it spilled through the gap between the tables into the monitor causing a cascade failure and a small fire which melted pretty much everything under the table. It took the company out for 2 weeks until Dell could ship new kit in.

There is a reason to do it properly IMHO (and experience!)