Over the years, we've come across a number of interesting hardware and software components. Some of them we've loved, some of them are worse than watching Initial D sober. As the list of do's and dont's grew, we decided to start listing them on a separate page instead of filling up the header of the server list page.
Due to the international nature of these technical details, this information is provided only in English. Feel free to contact us with any inquiries you may have.
Right here, right now
This part is for recent stuff.
After a number of tests with various SSD caching technologies, we've come to have an opinion on them in general: AVOID THEM! Naturally there are good options if you put serious money down, but we really can't go that way. In case you're curious, besides the usual major storage vendors, QNAP can provide good performance with 10GbE and SSD caching so for NAS solutions, I can give a general recommendation their way. When everything is weighed in, trouble vs cost vs complexity vs gains, we just came to the conclusion that all flash is the way to go for all primary data. We've started testing a radically new and as of yet very beta virtualization solution, which we'll make available to our users probably early next year.
There will still be places for spinning disks like certain mass archives, backups and such, but those places don't need extra speed. Keywords will probably be WD Red Pro, RAID-10 and iSCSI for those. We've still got the drawing board on the table.
Been a while since the last update, but not much has changed, not enough to write much. For the past half a year, we've been making the "better late than never" decision to move into Xeon CPU's at least for the about half our hardware base. The only meaningful reason is support for ECC memory. We're still running X99 series motherboards and I can't think of any good reasons to change that. Because we need our hardware to be easily adjustable to any use, we're not going to try any sort of more use type dedicated hardware (read: brand servers).
We have a serious practice of looking in the mirror about SSD's. Right now we use them for caching and booting and such, but at some point they're going to replace most of the primary data storage too. This year, next year, not sure yet. Some massive archives will still be on spinning drives for years to come, but that's not too many systems. Some insider calculations suggest that over 90% of our hosted virtual machines could run with inexpensive SSD solutions.
Our current hardware recommendations are Asus X99 motherboards, we've used a LOT of X99-Deluxe and X99-WS/IPMI. Both can be recommended warmly. On the storage front, WD Red / Red Pro / RE series get our thumbs up. We have insufficient data to heavily recommend any PSU models or other hardware at this point. We are happy to confirm that anything Seagate produces is still absolute garbage.
Haswell-E is here. Well, it's been here a few weeks but I never got around to updating this page. We're building two (for now) virtualization nodes from them and once the software looks stable enough, we'll start moving virtual servers to these new nodes, bit by bit. The nodes will get their names from the game Trinity Universe because fluffy ears.
The hardware? Yeah, looks like it's working. It'll be fun to test the new m.2 pcie ssd's at some point but I highly doubt we'll need those for virtualization nodes. The boot volumes just don't do much work.
Seriously, nothing much has happened for two years. The X79 based systems are still hot, CPU's have been upgraded to i7-4930K's but this base architechture is both rock solid and still very much up to date. We switched from Asus P9X79 Deluxe to P9X79 WS, but that's mostly cosmetic. They're both amazingly stable, solid motherboards, second to none in their class.
On the storage side, we're sorry to see Samsung leave the market, but so far we've been happy with WD drives. Seagate is permanently blacklisted due to catastrophical problems and failing warranty procedures. They're not bad, they're way worse.
Additionally, we found a few old IBM blades out of use and have gotten them to run some minor operations. This has freed some newer servers for inhumane experimental operations. No, you can't ask. Okay, you can ask, but we won't answer. At least not truthfully.
And speaking about uncomfortable truths, that old Albel is still running. Seriously, the machine is from the 90's. A modern toaster could run that workload.
That was fast. We managed to upgrade every i7-9x0 based system to i7-3820/3930K in under half a year, including the testing phase. Last week, we sold off our last i7-950 motherboard. Next, we're doing some major upgrades on several backup systems.
Christmas is finally here! That is to say, our first three Sandy Bridge E based systems have arrived, starting with i7-3930K. Epic testing starts now!
This year has been a hardware disappointment year. We started selling off older servers, mostly i7-860 and i7-920, starting spring this year. We figured that either AMD or Intel would bring something good to the market this year. We figured wrong. Bulldozer was a performance joke and Sandy Bridge E was a rather minor performance gain. On the other hand, the previously tested Sandy Bridge i7-2600 based setups failed due to lack of a suitable motherboard.
However, we have a major lack of hardware on our hands so we ended purchasing a few i7-3930K based setups, and expect to receive them early December. We'll be testing new virtualization setups and of course the usual stability tests before putting them into production use. Let's hope Ivy Bridge comes to the rescue next year.
The second major series of Core i7's has started its testing runs with a few i7-2600 setups. We're hoping the P8P67 EVO from Asus will work as fine as their P6T Deluxe series did. Also, we've stopped buying 2 GB DIMMs as 4 GB models have been priced more reasonably.
A second phase of virtualization testing started on 2010/06. The tests went on for a few months and moved into production in 2010/09 through 2010/10. After a large amount of testing, we found again that Xen was our best choice. Naturally the version was upgraded and changes to the management tools were tweaked as well.
Where it began
The information (and memory) is already a bit sketchy, but our first server was Youzen (youzen.bishounen.st at the time, reverse name pc3.vekoduck.com), originally a Pentium 133 MHz box running behind the desk at Kyuu's home on an unsupported test/development ADSL connection. The first OS used was Red Hat Linux version 6.2. That's something that none of us miss. The lowest end server on the B2 network was the first incarnation of Tamahome, which was an HP Vectra 486dx2-66 with 500+400 MB hard disks. It was actually upgraded step by step eventually into a Core2Quad Q6600, until we realized a virtual server was really all that was needed.
Testing out alternatives
The first production use non-x86 server was Lafiel (PowerMac 9500/200), put to use in 2002, upgraded to PowerMac G3 in 2005. In 2007 it was replaced by a G4 system named Rin. We also tinkered a while with Itanium2 hardware in the form of HP's 1600 and 2600 series Integrity rack servers. They were rock solid, fast, epic and in general quite cool, but didn't leave much room for affordable upgrades or tweaks. The same comments go for a SUN Netra we played around with for a while. Thus we went back to boring x86 and amd64 hardware. We do have a few Alix boxes doing some low power dirty work. Odds are we'll test a few of their APU's too. As for Raspberry Pi, we just haven't found any fun use for them. Still, we're keeping half a dozen around for philosophical reasons.
We've also tried out some brand name hardware now and then. In addition to the Apples and HP Integrity pieces mentioned above, there's been an ICL TeamServer, some HP NetServers and a number of HP Vectra and IBM workstations. With some exceptions, we mostly prefer the adventurous, renegade lifestyle that clone hardware brings. Some of us actually like screwing in motherboards on a friday evening.
More bits, more cores, more work
Since around 3Q/2004, 64 bit architechtures, starting with AMD64 were accepted as stable operating platforms. Unfortunately, 64 bit linux distributions lagged a bit behind, so it took almost a year to switch most of the servers to 64 bit operating systems. The first high reliability production server that switched to 64 bit hardware(AMD64) was Zelgadis, upgraded from P4 to AMD64 on 2005/02/22. The first production server to switch to a 64 bit OS was Tamahome, which switched over to Debian-Pure64 on 2005/04/15. During the same year, as more standardized Debian options became available, the upgrade became faster. Unfortunately before this happened, we had tested many not-so-supported 64 bit linux options and it took quite a while to reinstall them from scratch.
Dual core systems came around in mid-2005 and in production, August 5th 2005 for Tsukasa (later Haseo, which was retired as unneeded in 09/2009). We started with Athlon64 X2 models but very soon we found out that while AMD was faster at the time, the chipsets and motherboards just plain sucked. Eventually we started switching to Pentium D and soon after, around Q4/2006 to Core2 Duo, in Q1/2007 to Core2 Quad and in Q2/2009 to Core i7. Too bad for AMD - decent CPU's fail constantly because nobody could make a reliable motherboard (and yes, we've tested dozens of them over the years).
Virtualization tests were started in Q2/2007 and put to minimal testing soon after. We've been using Xen and Linux-vserver. So far, these seem to have worked nicely and larger production use started in late 2007. We've played a bit with Hyper-V and KVM as well, but these haven't been put to production use.
On the OS front, we've tested and haven't missed RHEL, Fedora or Gentoo. We've also played a bit with Ubuntu, but we've ended up preferring Debian for most uses. Some BSD testing has been tried, but there be dragons.
The coming of i7 and internal standards
Core i7 has quickly become the dominant architechture, starting with Youzen on 2009/05 and Lelouch (up to version 3) on 2009/06. First S1156 i7 on Zelgadis 2009/11 though we went back to 1366 soon after. Also during 2009/11, we're retiring the last Q6600 CPU's. In early 2010 we decided to emphasize purchases on a combination we've found to be foolproof: Asus P6T Deluxe V2, i7-9xx CPU, Kingston KVR memory and Nexus RX PSU's. By 2010/11, all servers had switched over to this setup. Most servers are also accompanied by a varied amount of Samsung F3 SATA hard disks.
The good experiences with the P6T based setups have emphasized the usefulness of standards. Instead of buying whatever hardware seems fun at the time, we'll buy certain models of motherboard, CPU series, HDD's and so on. This is most useful when dealing with hardware faults, when a replacement is always nearby.
Physical server numbers are decreasing. This seems like the natural trend to go with. We've dropped another two physical servers between Q4-2009 and Q1-2010. We'll probably go with this amount for now.
Starting 2009/06, all Core2 based dual core CPU's have been switched to quad cores in all production servers. The quad core Q6600 was a long standing classic for us, but the last one was finally retired in 2009/11. The last 775 socket CPU was retired in 2009/10, when we replaced the Q9550 on our co-operated server Athena with an i7-930.
We've also started adding hot swap SATA trays to new servers. With luck, they'll minimize downtime in the future.
This area is all about stuff we use, stuff we like and stuff we don't like.
Stuff we like
Nobody really makes the perfect computer case. However, Compucase has made some real effort with their 6919 series, which unfortunately isn't available anymore. With the 6919, putting pieces in and taking them out is quite effortless and simple. Anything can be changed without hurting your fingers or brains. No wonder we've stocked a dozen or so of these cases into a well guarded location. For rack use, we're going for the horrendously ugly UNC-410F-B. It's big, a bit expensive, looks like the 80's, but it's roomy, well cooled and easy to maintain.
A case needs a good PSU. We've found extremely fine products from Corsair's HX series. They're quiet, reliable, very powerful, providing amazing efficiency ratings, with the only downside of being a bit on the expensive side. We had a few Super Flower models, but their availability just sucks nowadays. Right now we're using FSP Aurum models alongside with Corsairs.
A good way to ease maintenance is with hot swap trays. We've used some from Icydock and from Chieftec. Nothing odd in either ones, they do the job.
Motherboards are a pain. In the stone ages, Intel was the only way to go, but for years we've gone with Asus models from the higher end (ie. stuff ending with "Pro", "Deluxe" or "WS"). They've worked very long, with very little trouble.
Hard drives are also a pain. Samsung's F3's were our tool of choice for years, but since they went off the market, WD is our brand of choice, with Red and RE models.
For memory, we'll buy Kingston. Haven't been able to break a single one yet. Noting that most other brands tend to break after a bit of component swapping and long term use, that's much said.
On the OS front, we're putting several thumbs up for Debian. It's not pretty, but it works and is stable. Once you learn it, maintenance is a breeze.
Stuff we don't like
Compucase may make decent cases but their PSU's (branded as HEC) aren't much to talk about. They just don't win in the efficiency/power output competition. They also tend to break too soon. Antec PSU's are to be avoided at all costs. Many Nexus PSU's have also proven to be less than ideal in reliability, though not the worst of the lot.
Motherboard makers need to get a grip. MSI boards tend to break all the time, so we forgot about them quickly. Actually, ASUS is the only one we can trust and even their cheapo models don't really create trust in us.
Purchase of new Seagate drives is blacklisted, their quality and warranty are both hideously horrible. We'll use what we have and never go there again.
Let's never use Ubuntu again, okay? I don't know what curse we have but every Ubuntu box has become an administrative nightmare. Reliability issues, odd breakages, random "let's remove this software without warning" decisions and such. Pretty odd, considering Debian is working fine and they're relatives.