Labor Pains

In some respects building an enthusiast class computer is akin to having a child: conceiving the child is a lot of fun, but the gestation and birth can be very challenging.

On Thursday, August 13 I finally got my computer case back from CoolIT Systems in Calgary. It looks like my photos and measurements paid off cause everything fit perfectly. It also helped that they put one of their best builders on the case - Sean Mutlow - because he really understood what to do.



Friday, the next day, I set forth to get everything working. This was my first time ever building a system from scratch - let alone something as exotic as this. I was mostly worried about installing the CPUs (at $1,500 a piece) and attaching the water blocks. When you attach a cooling system to your CPUs you have to use some kind of thermal paste to make a good thermal connection between the CPU heat spreader and cooling device. I had read a lot of articles and decided on Arctic Silver Ceramique as it showed the best conductive properties for CPUs. I had to remove the thermal grease from the water blocks that CoolIT has installed, then I had to prep the surfaces of both the CPUs and water blocks. I had to further prep the water block surface with Ceramique per the instructions, but it did not seem to make a difference as the surface of the water blocks was like a mirror. Finally a laid down a thin strip of Ceramique on the CPU and attached the water block. This was a bit tricky because the WS-240 cooler had all the hoses attached and it was awkward settling the water block on the CPU with the hoses trying to pull things ever which way.



After finishing what I considered to be the hardest part then came the tedious part of connecting all the power and data cables everywhere. This was extra challenging because the S5520SC is a big board and I really had to stretch some cables and connections.



Finally I had enough together that I could power on Gemini for the first time. This was rather distressing because after powering it on - nothing happened - there was nothing on my computer display to show that anything was working. I did this a few times and still nothing. Whenever I powered it down there was this beeping from the motherboard -which I discovered was the diagnostic beep codes. I looked them up in the manual and it said "DC Power Missing" so I fiddled with the power connectors a little and powered it on again. This time the boot sequence showed on my display and I breathed a sigh of relief. In hindsight I don't really know why nothing happened the first few times I power on Gemini, because every time I power it down I get the same beep codes.

Now that I was finally able to get into the BIOS settings I started exploring the disk system. To my surprise one of the disks was not working. Fortunately I had bought an extra disk as a spare so when I connected that one everything was ok again, and the ICH10R southbridge could see all 5 of my disks.

Next I tried configuring the RAID. I had thought this would be easy because I have configured RAID before on the ICH10R southbridge using the Intel Matrix Storage Manager, but this turned out to be a nightmare. It turns out that the S5520SC board does not use the Intel Matrix Storage Manager like every other civilized computer, but some other piece of crap called Intel® Embedded Server RAID Technology II (ESRTII). I design and implement computer user interfaces for a living and I can say with all expertise - the user interface on this utility is total crap. Let's just say that this whole process used about 2 hours of my time and involved a great deal of swearing.

After trying to configure the RAID I could not get Windows to install. I tried using the Intel Deployment CD to set up the RAID, and this was even worse crap. Then I made a bad decision. Because Windows would not installed I enabled a BIOS setting called "EFI Optimized Boot" and restarted Gemini. After that Gemini was a "brick" - that is it was nothing more than a very big paper-weight - it would start to boot - but would not even go into the BIOS setup. By the time Deena got home I was swearing like a soldier and bordering on a depressive mental break-down.

Friday nights is my night for Second Life - and lets just say that my technology woes only got worse that night as I ruined the entire beach. It took excessive quantities of alcohol and hard rock music to hang on to my sanity that night.

By Sunday I had had time to cool down enough to start thinking of solutions. My very good friend Remo (from Second Life) had suggested resetting the CMOS, so after searching through the manual I found a jumper on the motherboard that did that - and finally Gemini booted againg.

Eventually I gave up trying to set up RAID and switched back to a non-RAID configuration and finally got Windows 7 to install. Pretty well the first thing I did after that was run the Microsoft performance analyzer - which gave my a 7.8 out of 7.9 on my CPU and Memory performance. That was a very satisfying result.



My next big goal was to install Second Life and see how well it ran. Well it ran pretty crappy! Every two seconds there was a one second pause - everything froze. Needless to say I was getting pretty disillusioned by now. Not that I have children - but it's sort of like going through a challenging gestation and difficult birth - only to find out your child is mentally retarded. In spite of this limitation, I was actually able to run Second Life, which I cannot do any more on my Sony PCV-RX660; and Deena and I were finally able to be in Second Life together for the first time in 8 months. I was a hell of a lot of fun having her in-world at the same time as me.

For the rest of the week I played with various things, trying to learn more about the problems I had, and was still having. But all in all, it was very satisfying having my dream-system running after 3 years of planning, and 2 months of waiting for the case to be ready after initially ordering all the parts.

To be continued...

The Patience of Job

Well sometimes when you are the first person trying a thing, it takes time to get it right.



It took quite a while for CoolIT Systems to get my Boreas Chassis to me. I had asked them to install the Boreas chiller (for my GPUs), and their new WS-240 chiller for the CPUs. Not only is the 240 a new product, they had never installed one in this kind of chassis before, so things were a little cramped. After they got it installed they realized the fans were a little too thick - 25 mm. After some shopping around they finally found some low profile 20 mm fans.

All this took time to sort out but I finally got my chassis Friday, July 17. Since we had just moved homes the day before our new place was in quite a state of disarray - boxes and boxes everywhere. Finally Saturday morning I had some time and space to open the box and take out the chassis. I spent the morning installing the power supply, BluRay drive, and motherboard; but after installing the motherboard it was clear there were problems.

1. The coolant hose from the WS-240 was blocking the power connector for CPU-2.
2. The pump for the Boreas was blocking the main power connector.
3. The coolant hose between the two CPU water-blocks was too short.

In retrospect I probably should have shipped them the motherboard and these problems would have been obvious to them too. After talking to CoolIT on the phone on Monday they agreed to have me ship it back and correct the problems. They should get the chassis on Friday, and hopefully will be able to get it back to me soon.

While I am incredibly eager to finally get this computer system running, it has been over two years that I have been planning it, so if it takes a few more weeks to get it built right I have to be patient.

In the mean time rumors are the Intel will soon release a Xeon 5590 processor, which will likely be faster than the 5570 processors I bought. I have not opened the boxes my processors are in, so if Intel releases the new ones soon enough, I am hoping I can exchange my 5570s for a couple of 5590s.

Cheers, Eric

Obsolete the Day After You Buy It

So I've been waiting many many months for word on an Intel Skulltrail II motherboard. It figures that the week after I buy the Intel S5520SC motherboard I hear about the Skulltrail II

• Rumor says Intel Skulltrail 2 in the works
• Intel "Skulltrail 2" is based upon 8-Core Nehalem-EX

Of course there are no real specifications out yet, but it's interesting that they are planning it with the new 8-Core Nehalem-EX - that's 32 threads folks! We'll be lucky to see the Nehalem-EX before 2010.

I'm really torn because the Skulltrail II was what I really wanted, but then do I want to wait another 6 months? Also, the Skulltrail I was a terrible motherboard - I don't know what Intel was smoking when they came up with that. They got so much criticism I can only hope they learned their lesson and won't be so stupid next time. I know deep down in my heart I could design a really cool Skulltrail II - probably better than Intel.

So where does that leave me? If the Skulltrail II incorporates an ICH10 compatible RAID controller I would be able to swap out my S5520SC with a Skulltrail II and preserve my RAID file system. Hopefully there would be a spare 4/8 lane PCI Express slot for my FusionIO ioBoard. Of course I would have to by new processors - the EX ones. I suspect the EX processors will not be socket compatible with the Xeon 5580s, so I can't just drop them into my S5520SC motherboard.

What else does Skulltrail II get you? Well probably the ability to overclock the processors. It is well known the Nehalem processors can easily handle 4 GHz or better.

Anyway - so many if's right now. My current plan is to proceed with building Gemini and 6 months to a year from now if the Skulltrail II is appropriate, maybe we'll see Gemini on steroids :-)

Getting it all together

Last Friday I started receiving the parts for Gemini. Pictured here from top left to right is

1. Enermax Galaxy 1250 Watt power supply
2. Intel S5520SC motherboard
3. ATI Radeon 4870 graphics card (temporary)
4. LG 50 GB Blu-ray reader/writer
5. two Corsair Dominator 6 GB 1600 MHz memory kits
6. two Intel Xeon 5580 processors
7. Intel RAID 5 activation key for the S5520SC
8. 250 GB HD (temporary) for testing
9. ArctiClean Thermal Material Remover
10. Arctic Silver Ceramique - thermal paste
11. ArctiClean Thermal Surface Purifier
12. Sun Microsystems keyboard and mouse
    

I was able to get all these parts from NCIX locally (except for the keyboard) and they gave me a nice discount too.

I could not find any keyboards I liked anywhere, but I have an old Sun Workstation that still has one of the better keyboards I have ever used. However it's not compatible with personal computers. So I ordered one of Sun's newer keyboard kits that is compatible with personal computers. Now if only I could find some way to get all the keys to work with Windows.

The last piece I'm waiting for - the biggie - is my Silverstone TJ-07 case with a CoolIT Systems Boreas and Domino WS 240 water cooling for the graphics cards and CPUs. Then I can finally assemble the first phase of Gemini.

On back-order I have 6 Western Digital Enterprise 2 terabyte disk drives coming. They are so new they are still a month or two away, and in the mean time I have a smaller 250 GB drive I can use for testing.

I've already got a copy of 64-bit Windows 7 Ultimate Release Candidate. When Microsoft finally release Windows 7 in October, I'll go get a legitimate copy then.

I'm also planning to get a Dell UltraSharp 3008WFP 30'' Widescreen LCD Monitor - but I can get that just about any time. I may as well spread out my spending over a few pay cheques.

In the next phases I plan to add a FusionIO ioDrive, but I'm still waiting for them to announce when you can boot from the device. That's going to be my boot drive and it's going to be blazing fast.

In the final phase I plan to get two ATI Radeon 5870 X2 graphics cards, and two Koolance water blocks to connect to the Boreas water chiller. Hopefully these will be ready sometime this fall. I will be ready for some intense GPU overclocking. Also, these cards will be the first to support DirectX-11, the new graphics standard in Windows 7.

We plan to move in about a month to a slightly larger place. One of the things I intend to do is have a dedicated 15 Amp, 240 Volt circuit added to the second bedroom just to power Gemini. It will be a great way to warm the room in the winter. At any rate, it will be interesting to see if I finally get Gemini powered up before we have to move.

When I get things working relatively well I will probably have a coming out party for Gemini.

Recipe for a Well Baked Computer

When most of us buy a computer, generally some bright person somewhere has designed the components to all work well together - but this is not strictly true, I've seen some pretty half-baked computer systems the came off the store shelves.

Designing a computer system is really an exercise in compromise. A good system will have the fewest compromises possible, and all the components will work well together as a whole. Some of the main factors I didn't want to compromise on were: motherboard, graphics cards, file system, power supply, and cooling system.

Power Supply - I really did not want to some day add another component to my computer system and find out there was not enough power available. In my case I ordered an Enermax Galaxy EVO 1250 Watt. This is about the most powerful power supply you can get for a tower PC system. In particular it it has a 24-pin connector for a high-end motherboard like mine, and two 8-pin connectors for each CPU, again for a motherboard like mine. Additionally it has more than enough PCI Express 8-pin connectors for two top-end graphics cards, and the cooling system I want. It has way more disk drive connectors than I will ever need. Finally, this power supply has a really large fan, which means it is relatively quite.

A funny thing I discovered. The power supply I really wanted was the Enermax Evolution 85+ 1250, but it is not sold in North America. This is one of the most efficient power supplies available - over 85% efficient (i.e. wastes less power). For some reason they could not get it approved in North America, so they took the very same power supply, and tuned it down to about 82% and then they were able to get it approved in North America. Go figure? I actually could have gotten the Evolution 85+ from Australia, but it only supports 220 - 240 Volts, and that can be hard to find in Canada in some places. On the other hand the Galaxy will run from 115 - 240 Volts which gives me more options. I still plan to run with 240 Volts, because that is more efficient than 110 Volts, but who knows where I will end up or where I will want to take my computer to.

File System - Most PCs these days have one disk drive, generally with a C: and a D: partition. Historically the C: partition is for your operating system, and the D: partition is for your data files. In some computers C and D are actually separate disk drives. In general you want your C drive to be as fast as possible because that is where the system boots from - and I hate waiting for my system to boot, or reboot. Also, that is typically where you run all your applications from, so again you want it to be fast. Finally, often people run their page-file on the C: partition, and that needs to be fast too.

One thing I have noticed recently is that some system builders use a really fast drive for their C, like a Western Digital VelociRaptor or an Intel X25-A SSD. These drives sacrifice storage space for speed. Then people will use a slower but much larger disk for their D drive. This seems like a good strategy. Some extremests will actually run a RAID 0 array of SSD for some intense speed.

More recently a new technology is becoming popular, sold state memory on a PCI-Express card. The beauty of this technology is that you can get the same performance as a RAID 0 array of SSD (or better), but all on one PCI card. Currently my favorite is something like the ioXtreme from Fusion IO. My strategy is that I would use this for my C drive, and then use a big RAID 5 array for my D drive. This would give me the best of both worlds - honking fast boot, reboot, paging, and application start-up; and massive storage for music, pictures, videos, etc. The only draw back is currently you cannot boot from a Fusion IO board, but they are working on it.

As I plan to build my system in phases I will build the RAID 5 array first and boot from it. When I get a bootable Fusion IO board I will then switch to booting from that. Additionally, I plan to backup my Fusion IO board to my RAID 5 D drive - so everything fits together nicely. I would even keep my RAID 5 array bootable, for emergencies.

Graphics Cards - 4 Graphics Processing Units (GPUs) is about the maximum you can put in a computer. Fortunaly ATI/AMD make some graphics cards with 2 GPU per card. In particular I'm waiting for the new Radeon 5870 X2 to become available (they were just announced recently). These cards use a lot of electrical power - which is why I ordered the power supply I did. All that power has to go somewhere so these cards generate a lot of heat. My plan is to water cool the cards using a CoolIT Systems Boreas chiller. This thing is actually like a mini refrigerator because it uses thermocouples to make the water very cold, consequently removing even more heat from the GPUs. This will come in very handy if I want to over-clock my GPUs. Since the GPUs I want are not for sale yet, I've decided to go with a more modest GPU during phase one of building Gemini.

Motherboard - I've already talked about the Intel S520Sc motherboard I've ordered so I won't go into to much more detail except to say finding a way to cool the CPUs was tricky. Originally I was going to use the Boreas to cool the CPUs, but this motherboard does not support over-clocking, so the Boreas would sort of be wasted. Lucky for me CoolIT Systems just came out with a new water-cooled radiator, the Domino WS 240. The advantage of this is that it will give the CPU better cooling than stock fans, and it fits in the top of the computer case, and leaves the Boreas for some hot GPUs.

The Need for Speed

If computers get faster and faster every year, why do we seem to spend more and more time waiting for them to do things?

The problem is software. People have an insatiable need for more and more features - bells and whistles. Software companies are driven to add more and more features, more and more cool things. Each new software application, or each new version of the same application tends to get bigger and bigger, and bloated with more features the slow down the computer.

For example, web browsers! When the web first started the first web browser, Mosaic, was fairly simple, it could display text and pictures and it didn't require a lot of computer power to do that. Over time people have added other multimedia such as videos (i.e. You Tube) and scripts. These take up more and more computer power. Now that there are tabbed browsers it is common to have many tabs open, and these may all be running scripts or playing media. Increasingly more and more web sites use scripts and media to add the cool factor, or get people's attention (i.e. advertising). When I start up my web browser now, it puts an awful load on my computer slowing it down. All those tabs, media and scripts cause the browser to use more and more threads and processes - so you need a computer that can run a lot of threads and processes at one time.

Another thing people tend to do, especially me, is have lots of applications running and windows open. The more windows you have open, the more memory is used and the more load there is on the computer. The more software you install the more chance there is that the software starts up automatically at boot time, so whenever you reboot your computer it can take longer to finish booting. My current computer takes about 15 to 20 minutes to actually finish booting. Sure I can login a minute after booting, but everything runs incredibly slow until it's finished booting. Finally, a sad fact about Microsoft Windows is that for some reason we seem to always have to be rebooting it.

The bottom line is when the computer gets too slow it becomes more frustrating to use it. The only way to reduce the frustration is to get a faster more powerful computer. In my case my home computer is for my own personal enjoyment - that's the worst time to be frustrated.

A Tapestry of Threads

In the early days computers could only execute one instruction after another, there was a single CPU and a single Thread of execution. Computers give the illusion of doing many things at once by a technique called Time-Sharing: the work is divided up into many Processes, and the Operating System runs each process for a few thousandths of a second, switching from one process to another. Several people could be using a computer at the same time, each working under the illusion that they had the computer all to themselves - except the more users and processes there were the slower the computer seems to get.

For a long time computers have gotten faster and faster, but at every moment in time computers were just as fast as they could go. The only way to make a more powerful computer was to tie two or more Central Processing Units (CPUs) together in one system. In most cases all CPUs shared the same memory, and the Operating System could take any process and run it on any CPU, making the whole system more powerful because now more than one process could be running at the same time. This was especially useful on systems with lots of users. These were the haydays of mainframe computers.

When personal computers became popular there was never a big need for more than one CPU because each person had the whole system to themselves. For a long time CPUs just got faster and faster every year and were able to do more work, keeping up with increasingly sophisticated and demanding software. About 5 years ago a terrible thing started happening, CPUs stopped getting fasters, primarly because we reached certain limits in the laws of physics. For at least 5 years the fastest CPUs are about 3 to 4 GHz and that's just not likely to change with current electronic computer circuits. The fastest a computer has run to date is just over 7 GHz, but that requires liquid helium and is very expensive and fragile.

But people still demand ever more powerful computers every year. They only way computer makers have been able to make systems more powerful is by adding more CPUs. Now the terminology gets a little crazy. In a single chip computer hardware people refer to the chip as the CPU. Chip makers actually put two or more CPUs on one chip, but they call them Cores. If a CPU has 4 cores the operating system views this as 4 separate CPUs, and software people call them logical CPUs. Some CPUs use a technique called Hyperthreading where each CPU or core is able to run two or more separate threads of execution at the same time. Hyperthreading is not as powerful as adding more cores, but it is an economical way to run more than one process at a time. So if you have a chip like Intel's Core i7 or Xeon 5500, Intel calls this a Hyperthreaded 4-core CPU - but to Windows this looks like 8 separate CPUs. In fact if you open Windows Task Manager you will see 8 CPUs.

Now the system I am building is a Dual Xeon 5580, to Intel that means two CPUs, 8 cores, and 16 threads. To Windows that means 16 CPUs.