To run Matlab persistently (so it doesn’t die when you log out), “screen” can be used as described at this link.

To run Matlab non-interactively, I found a concise and useful page with instructions and explanations: http://people.scs.fsu.edu/~burkardt/m_src/matlab_batch/matlab_batch.html .

Email and collaboration tools

Dartmouth’s Council on Computing has constituted a task force to determine Dartmouth’s requirements for future email and collaboration tools. Thayer School’s Director of Computing Services is on the task force and seeks your thoughts on the subject.

Hard drive based back up server

We are a “belt and suspenders” computing staff. To reflect our paranoia of making sure all your ThayerFS data is safe, we’re adding another layer of redundancy to our data back up plan. We just started setting up a new “online backup server”. It consists of fifteen 1 Terabyte hard drives which we’ll use to back up ThayerFS. This system will eventually be located off-site. We’ll continue to use our tape library which will be located in yet another off-site location.

Weather station and Solar Panel monitoring

We’re in the middle of a project to get a weather station installed on the roof of Murdough. We’ll be using the same system to monitor the output of the Solar Panels which are already installed on the roof. The plan is to make the data available on the web for those interested in local conditions and for research purposes.

Spanos lighting improvements

We have been working with our building manager and FO&M to add special lights to properly illuminate presenters. This should improve the quality of our lecture capture video and should reduce the harsh shadows the current lighting causes on the speaker’s face.

A/V system improvements, documentation, and repairs

Collaborating with Instrument Room personnel, we have been working for some time now to make our classroom and meeting room audio visual systems more user-friendly. This includes simplification of touch screen controls or even their replacement with pushbutton controls, enhanced documentation about how to use the systems, improved image quality in Spanos and C200, a combination white board/projector screen in M210, additional microphone options, improved/repaired audio, etc. Some of these improvements are finished, but we still have a long way to go on others.

New Intel compiler available

We purchased two floating licenses of Intel Fortran and C++ compilers for Linux. They are available on any of our Linux clients or compute servers. Also included with these licenses are the MKL and IPP libraries.

For more information about these new compilers and how to use them, please see our Linux Services page.

New lab computers

In the next few months, we plan to deploy new lab computers for the Linux lab in Cummings and to replace the ten oldest computers in MacLean 210.

Adobe Contribute rollout soonish

We’ve purchased licenses of Adobe Contribute, which will allow the Thayer Community to create and edit web pages in a simple WYSIWYG editor. The experience is similar to a word processor. If you are interested in using this software, please contact us.

Vista planning

While we’re content with Windows XP, Windows Vista is an inevitability as XP support is dropped by Microsoft. We’re going to start looking at the best way to image and deploy Vista on lab, desktop, and laptop computers.

Blade servers

We recently took delivery of a new blade chassis and some new blade servers. The chassis can accommodate up to 16 servers. The preliminary plan is to replace our aging babylon compute cluster with a couple of blades. This will reduce space, electricity, cooling, and administration time, while increasing the computing power over our current cluster. We’ll have several empty slots available for faculty with research projects that need extra computer power.

Condor High Throughput Computing

We’re investigating the use of Condor High Throughput Computing on our linux clusters. This would allow the Thayer community to submit compute jobs and have them distributed automatically to our compute clusters.

Currently, we are testing this on our limited-access sisyphus cluster, but hope to roll it out to an upgraded babylon cluster after the Spring term. Jobs run on either cluster will be able to use the other cluster’s CPUs if they are available.

Hardy Heron is coming…

The new version of Ubuntu, version 8.04 (hardy heron), is scheduled to be released at the end of April. This new release features many enhancements to the version we’re currently running on our linux clients, and is also the next of Ubuntu’s “long-term release” versions, which will get security updates for a longer period of time than their other versions.

We have been alpha (and now beta) testing this to ensure that any bugs related to our systems are reported and corrected prior to its official release. We plan to upgrade the Cummings 227 linux lab with the new version (and new computers, too!) after the Spring term. We will also be in touch with faculty and staff who have linux clients to schedule their upgrades.

ThayerCups server upgrade

We just upgraded the CUPS print server that handles print jobs from Mac and Linux clients. There should not be any change in the way you print from these platforms.  The new server is now easier to back up.

Application Virtualization

We are testing, “Application Virtualization”, a new technique for encapsulating applications to ease deployment to lab computers and client computers.

Atrium Help Desk continues

We continue to hold our “Atrium Help Desk” from 3:00-4:00 on Tuesdays and Thursdays. Come by the atrium if you have any computing-related questions.

On Friday the 8th, a hard drive failed on one of our physical servers. The server had four production virtual servers on it. Supposedly, the hard drives we use in our servers are less likely to fail. They come with 5 year warranties, and graduated at the top of their class. To be on the safe side, each of our servers have 2 hard drives that are mirrored. So if one drive fails, the server just keeps operating without a hiccup.

The story could have ended there. We could have just left it until Monday, ordered a replacement drive, and installed it without any downtime. The likelihood of the second drive failing in the next few days is low, and we have backups. However, what if the other drive did fail? That would mean downtime as we reinstall and restore data… and downtime means sad customers (you).

Don’t worry, we’ve got a backup plan for our backup plan. Actually, we’ve got many layers of backup plans that could put you to sleep. Here’s the quick version of contingency plans that actually came into play.

We use Nagios (set up by Sys. Admin Matt Dailey) to monitor all our systems and let us know if something is unhappy. Matt also installed utilities on the servers for keeping track of the hard drives, and other critical hardware. If a drive gets grumpy, Nagios knows, and immediately sends us an email. And for critical things like drive failures, sends us a text message to our cell phones.

I received the text message and after verifying the problem, called Jordan (who happens to be the primary maintainer of the virtual servers in peril).

Jordan immediately transferred the virtual servers to a backup blade server we had waiting on standby. Even over a slow dial-up connection, he was able to do this from home.

The next day, Saturday, Matt Dailey came in to replace the bad hard drive with a spare we had on hand for just such an occasion. Matt pulled the bad drive out, and inserted the new drive. The good drive immediately started mirroring all its data to the new drive for the next time something goes wrong.

Everything is rosy!

Well mostly rosy. Considering this was our first disk failure since I’ve been working here, everything went surprising well. However, there are always surprises. Thankfully, none of the ones we encountered caused any hardship to the community.

First, when you plan for disaster, it is often easy to not cover the basics…. like… where the spare hard drive actual is located. We all were sure that we had put the spare drive on top of the servers. However, we installed larger capacity drives several months ago and the spare never made it to the server room from our office. Thankfully, Matt was easily able to find it.

Second, it is a pain, but it can be really helpful to simulate failures. It took the server 11 hours to fully mirror a 147 GB hard drive. This is much slower than we anticipated. Because we had already moved the production virtual servers off the system, it wasn’t an issue. We have been told by Dell that this is a limitation of the RAID controller. So this is a limitation that we’ll need to work around.

The third hiccup we’ve run into is that Hitachi no longer manufacturers the hard drive we are using. I purchased a new spare from Seagate that has all the identical specs. However, for reasons Dell doesn’t even understand, the RAID controller refuses to allow two drives from different manufacturers to work together. For systems that you plan to have in production for many years, buy more than one spare, or at least know that you may need contingency plan when you can no longer buy a specific component.

While the change isn’t very visible to the community, for us Systems Administrators (Jared, Jordan, and Matt Dailey), the change is significant.

Virtualization allows us to run multiple “computers” on one physical computer. So we can easily run independent servers for web site, email, databases all on one real server.

The short story is that we now have over a dozen virtual servers in production which has allowed us to save space, lower our hardware expenses, reduce electrical and cooling usage, ease backup, and improve reliability.

Read on for more details of our move to virtualized servers.

Setup details

Virtualization Software

We briefly looked at User-Mode-Linux, Xen, and more recently KVM, but in the end decided to use VMware server (not ESX). At the time, VMWare had recently released the free server version. We found it’s setup and administration to be very straightforward, and VMWare’s technology was mature. We decided to move cautiously. First experimenting with the product, then moving a server that was non-critical. Once it proved stable, we converted more and more of our servers to virtual machines. We’ve now gone through several iterations of server hosts and come to a solution that is stable and flexible.

Space

Thanks to our new server room in MacLean, space wasn’t at huge premium. But is definitely nice to have your servers consolidated. The reduction in cables alone is huge.

Cost

Costs of server hardware can be dramatically reduced. Many servers sit idle all day. This means one physical server can run many virtual servers. We’ve decided to run no more than about 4 virtual machines per physical server. However, there are several other hidden costs which are reduced. Less physical servers means less cooling and less electricity consumption.

Backup

The flexibility of backups that virtualization provides is one of the biggest changes from the move. In the past, we’ve used many techniques to do full backups that, in the event a physical server blows up, would allow us to quickly restore the server from scratch onto new hardware. Tools like dd, partimage, udpcast all work fairly well. However, they require the machine be shutdown and automation can be tricky.

With the help of LVM and a few other basic utilities, we have successfully implemented a completely automated full server backup system. The backup procedure is as follows:

• Pause the virtual server
• Create an LVM snapshot
• Resume the virtual server (steps 1-3 require about 3-15 seconds of downtime)
• Copy the Virtual machine files to the backup server
• tar and gzip the files for safe keeping and space savings

For us, 3-15 seconds of downtime during non-peak hours is acceptable, so we create these full backups for all virtual machines every night. Once the files are compressed, the disk usage per server is typically very small 500 – 1500 MB. We keep several nights worth of these backups in case the most recent backup has corruption.

On many of our Virtual Servers, we continue to do rsync backups of important directories from within the OS. Initially, we did this so that, in case the LVM snapshot method wasn’t robust, we’d have a fall back. However, it is sometimes handy to access just a single backup file (ie. a config file from /etc), so we’ve kept them around. Disk space is cheap, and you can never have too many backups.

Reliability

Instead of spending all your server budget on many quasi-fault-tolerant servers, with virtualization, you can buy fewer very fault-tolerant servers. We have a blade chassis with 10 server blades which we run our virtual servers on. The blade chassis has four power supplies, and each blade has two network interfaces (which we’ve bonded for failover), and two hard drives which are mirrored. The system is connected to different electrical circuits and two different switches. If one blade begins to fail, we can quickly copy the virtual machines to another host (this takes approximately 10 minutes per virtual machine, could be faster if we were using a SAN). If the entire blade chassis were to melt down, the Virtual Machines can be restored to any machine running VMWare server. With the release of VMWare Fusion for Mac OS X, we could even run our critical servers off our MacBook Pros :)

Testing

Whenever you do a server upgrade, there is a chance something will break. Virtualization offers us a couple nice safety nets. VMware supports snapshots of the VM at any given point, which can be restored. We can also restore to a LVM snapshot backup. There is also the option of creating a new development virtual machine, testing the change, then applying it to the production server. When you are done with the development server, you can simply remove it. All without touching any physical hardware.

The down sides

While virtualization has been very positive overall, there are a couple down sides to keep in mind.

The biggest issue is that all your virtual machines are dependent on the host server. If your host machine crashes, it means all virtual machines will be down. Early on, we ran into a serious LVM bug in the Ubuntu 6.06 stock kernel. Creating and removing snapshots caused kernel panics on the host. We made the decision to use a minimal Ubuntu 6.10 as the host OS and haven’t had issues since.

Kernel upgrades also require some forethought. VMware can be set up to automatically shutdown and start up the virtual machines when the host machine is rebooted. However, VMware kernel modules need to be rebuilt whenever the kernel version changes. It isn’t complicated, but it does require some planning ahead.

Virtual machine performance can be slower than real servers. Our servers are rarely under any significant load, so we haven’t run into this yet. However, it is something to keep in mind. File servers and database servers are probably left as real servers unless they are very lightly used.

Long time passing

From all the buzz in the IT world regarding virtualization, it is clearly here to stay. As management tools continue to improve, I’m sure we’ll be re-evaluating the offerings, especially from the open source community. Until then, we are extremely pleased with our implementation.