There are many reasons to pay for the reliability, rich feature set, and fail-over capabilities of our NetApp. The NetApp is an excellent product. However, there is also a significant amount of data at Thayer that just doesn’t need all the advanced features. For instance, we have research groups with multi-Terabyte data sets that just want their data easily accessible. If it were unavailable occasionally it wouldn’t be a huge deal. Other features such as snapshots are not critical.

If we can store these larger data sets on a cheap storage alternative, we can minimize the size and cost of our NetApp.

Design philosophy

After stalling for a couple years, hoping something would come along, we have decided to move forward with a plan that makes many compromises, but should meet the basic requirements for many of our storage needs.

We initially started the search to replace the NetApp altogether feature for feature. However, over time we lowered our standards to requirements that meet most of the needs for simple storage of lots of data. As our research continued, our philosophy on what product to choose evolved. In the end, the three main philosophical items we looked for when choosing a cheap storage solution were:

1. Widely used technologies and standards – If lots of people are already using the technology it is more likely to be stable. It also is likely to be around in a few years. While little start ups may have cool technology, we really don’t want to trust our data to a half-baked product. Little companies can also get gobbled up or go out business overnight, never to be heard from again.
2. Healthy communities and companies that are committed to the technology – In case we do have questions, or run into an issue, we want an active community of people that will be able to knowledgeably answers our questions. It isn’t enough that the technology says it supports CIFS and Active Directory. If it doesn’t work quite right in _our_ environment, and nobody seems to know why, we just can’t be comfortable moving forward.
3. Simplicity – Simple solutions have less chance of breaking and are easier to fix in the event that they do break. It is easy to build several layers of redundancy, but end up with a system that is more likely to break because of all the moving pieces.

What we considered

ZFS on OpenSolaris/Solaris

Link

We actually own a Sun Fire X4540 (Thor) running Solaris. We are currently using it as our online backup for the NetApp. It seems like a nice piece of hardware and the ZFS feature set is amazing. Snapshots, filesystem compression, expandability, checksumming, and the rest are all great.

• Technically a home run.
• Doesn’t score so well on our three design philosophy requirements.
• Reasonable, researched questions in forums go unanswered.
• Sun Engineers try to be helpful, but can disappear for weeks at a time.
• Sun shunned the Samba project and instead decided to implement their own CIFS server.
• Projects lack focus. There are several different editions of Solaris. They appear to be attempting to clear this up, but I really wish their “Project Indiana” broke free of a lot of the legacy issues weighing them down.
• Storage systems must be purchased fully populated.
• We could not get AD group integration working and ID mapping is complex.
• Even the latest build of OpenSolaris makes you feel like you are a Sys Admin from 1999.
• Sun laid off many employees. They have lots of projects they are trying to maintain, but seem to be having trouble keeping up with them.
• To make things worse, they got gobbled up by Oracle. How important Solaris is to Oracle remains to be seen.

Sun Storage 7000 Unified Storage (Amber Road)

Link

Sun realized there were admins like us who wanted ZFS but didn’t want to admin Solaris. So they wisely made a storage appliance. They did a great job of putting a pretty face on top of ZFS and Dtrace and then went and charged way too much for it.

• Price is lower than NetApp and EMC, but not low enough to build a decent sized userbase and build some critical mass.
• Can’t replicate from an Amber Road device to our existing Sun Fire X4540.

ZFS on Nexenta

Link

Nexenta seemed like it could be a good compromise for some of our Solaris concerns. Like Amber Road, it hides all/most of the Solaris administration and allows you to use ZFS like an appliance. We set up Nexenta on test systems on multiple occasions and it shows a lot of promise. Again a home run technically, but falls apart on our design philosophy.

• The company is very small without a very big user base.
• If you have any Solaris specific technical problems, they basically say, “go talk to Sun”.
• The depth and breadth of their documentation needs some major improvements.
• Fairly inexpensive (good educational discounts) but when building cheap storage, every additional dollar pushes up the cost per gigabyte.
• They are working like mad to add advanced features like failover.

Came very close to giving them a shot, but they just need a little more time to fully bake.

Clustered Samba

Link

I hope this project gains momentum as we are really drawn to an open system that we can grow on demand by adding another server full of hard drives as our storage needs grow in fits and starts.

• Still early and very little documentation
• Uses clustered file system which are still maturing and poorly supported on Ubuntu

Netgear ReadyNAS

Link

Netgear sells the ReadyNAS, a decent Linux based Network Attached Storage. We bought a 6 bay ReadyNAS Pro which we currently use for network based Time Machine backups for a few Macs. Bascially we wanted to get some mileage with a Linux based NAS.

• Decent web management interface with easy AD integration.
• Seems to have better throughput than most small business NAS systems.
• Tech Support is friendly but very bureaucratic. We ran into a bug when trying to add user and group permissions to a share. Getting tech support has been very difficult. You basically have to give them your first born child in order to be raised to the higher level of support. They really need to say, “yes that is a bug, we’ll fix it and get back to you”. Instead, they had me do dozens and dozens of unrelated tweaks to no avail. When I was finally suppose to get the next level of tech support, my ticket somehow got closed and I basically had to do the dozens of task all over again under a new ticket (which is still unresolved).
• The largest product they sell is 12 bays. Probably not big enough for us unless we used multiple systems.
• It uses EXT3 filesystem, so quota is user and group based instead of directory/project based. (can be worked around with lots of groups).
• Dollar per gigabyte is very good (currently around $0.53 per usable GB for the 3200 with 12 2TB drives in RAID 6 with 1 hot spare).

Btrfs

Link

A new, GPL filesystem originally developed by Oracle. All the fancy features that we want, baked into the Linux kernel. Fingers crossed that in a couple years this will have stabilized to the point that it is ready for production use. It seems like it has the most promise of competing with NetApp’s WAFL and ZFS. Oddly enough, Btrfs is now owned by Oracle. Will Oracle continue to develop both ZFS and Btrfs?

Other

We looked at numerous other technologies in varying levels of detail. None seemed to quite have the momentum at this point, and several were just not designed for a typical file server. We’ll keep an eye on them.

Some of the projects include:

• IBRIX – now owned by HP
• LeftHand Networks – now owned by HP
• Cleversafe
• Parascale
• Caringo CAStor

Read part three… what we actually chose to go with (warning, it is pretty anticlimactic).

The cost of our NetApp storage is a little fuzzy. Over the years, we’ve come up with calculations from around 2.50 – 12 dollars per gigabyte. But if you head over to NewEgg and buy a stand-alone 2 terabyte enterprise SATA drive, it is only $0.16 per raw gigabyte. Obviously there is a fair amount of additional hardware and software required to make that drive available on the network. So the question is, how can cheap can make a file server that still has respectable features, availability, and performance.

We’ve planned a system that comes out to about $0.28/GB raw and $0.37/GB usable. If the system works well, our plan is to buy a near identical system that we’ll use for backup. So backed up, the cost will be double, or about $0.84 per usable gigabyte. While this is many times more than the cost of a raw drive, it is significantly cheaper than the alternatives that we explored.

Over the next few blog posts, I’ll give some details of the options we investigated, what we’ve ultimately decided to go with, and how the implementation actually works out.

Update: The original price per GB I used was assuming we would buy a spare part of almost every electronic component (motherboard, CPU, RAM, etc.). I’ve updated it to reflect the price without any spares parts, which is still slightly misleading, as we will buy some spare parts. The price is also based on a fully populated system. However, we will initially only have a system that is 25% populated. At the beginning, this pushes the $/GB up, but because $/GB of hard drives is going down over time, by the time the system is fully populated, it will actually be lower than above quoted price.

Go to part two of the Cheap Storage Project series.

The “no questions asked” maximum quotas for graduate students remains at 5GB and 10GB for faculty and staff. As always, if you have reasonable case for needing more space, just let us know.

Full details about ThayerFS quotas are here.

Email and collaboration tools

Dartmouth’s Council on Computing has constituted a task force to determine Dartmouth’s requirements for future email and collaboration tools. Thayer School’s Director of Computing Services is on the task force and seeks your thoughts on the subject.

Hard drive based back up server

We are a “belt and suspenders” computing staff. To reflect our paranoia of making sure all your ThayerFS data is safe, we’re adding another layer of redundancy to our data back up plan. We just started setting up a new “online backup server”. It consists of fifteen 1 Terabyte hard drives which we’ll use to back up ThayerFS. This system will eventually be located off-site. We’ll continue to use our tape library which will be located in yet another off-site location.

Weather station and Solar Panel monitoring

We’re in the middle of a project to get a weather station installed on the roof of Murdough. We’ll be using the same system to monitor the output of the Solar Panels which are already installed on the roof. The plan is to make the data available on the web for those interested in local conditions and for research purposes.

Spanos lighting improvements

We have been working with our building manager and FO&M to add special lights to properly illuminate presenters. This should improve the quality of our lecture capture video and should reduce the harsh shadows the current lighting causes on the speaker’s face.

A/V system improvements, documentation, and repairs

Collaborating with Instrument Room personnel, we have been working for some time now to make our classroom and meeting room audio visual systems more user-friendly. This includes simplification of touch screen controls or even their replacement with pushbutton controls, enhanced documentation about how to use the systems, improved image quality in Spanos and C200, a combination white board/projector screen in M210, additional microphone options, improved/repaired audio, etc. Some of these improvements are finished, but we still have a long way to go on others.

New Intel compiler available

We purchased two floating licenses of Intel Fortran and C++ compilers for Linux. They are available on any of our Linux clients or compute servers. Also included with these licenses are the MKL and IPP libraries.

For more information about these new compilers and how to use them, please see our Linux Services page.

New lab computers

In the next few months, we plan to deploy new lab computers for the Linux lab in Cummings and to replace the ten oldest computers in MacLean 210.

Adobe Contribute rollout soonish

We’ve purchased licenses of Adobe Contribute, which will allow the Thayer Community to create and edit web pages in a simple WYSIWYG editor. The experience is similar to a word processor. If you are interested in using this software, please contact us.

Vista planning

While we’re content with Windows XP, Windows Vista is an inevitability as XP support is dropped by Microsoft. We’re going to start looking at the best way to image and deploy Vista on lab, desktop, and laptop computers.

Blade servers

We recently took delivery of a new blade chassis and some new blade servers. The chassis can accommodate up to 16 servers. The preliminary plan is to replace our aging babylon compute cluster with a couple of blades. This will reduce space, electricity, cooling, and administration time, while increasing the computing power over our current cluster. We’ll have several empty slots available for faculty with research projects that need extra computer power.

Condor High Throughput Computing

We’re investigating the use of Condor High Throughput Computing on our linux clusters. This would allow the Thayer community to submit compute jobs and have them distributed automatically to our compute clusters.

Currently, we are testing this on our limited-access sisyphus cluster, but hope to roll it out to an upgraded babylon cluster after the Spring term. Jobs run on either cluster will be able to use the other cluster’s CPUs if they are available.

Hardy Heron is coming…

The new version of Ubuntu, version 8.04 (hardy heron), is scheduled to be released at the end of April. This new release features many enhancements to the version we’re currently running on our linux clients, and is also the next of Ubuntu’s “long-term release” versions, which will get security updates for a longer period of time than their other versions.

We have been alpha (and now beta) testing this to ensure that any bugs related to our systems are reported and corrected prior to its official release. We plan to upgrade the Cummings 227 linux lab with the new version (and new computers, too!) after the Spring term. We will also be in touch with faculty and staff who have linux clients to schedule their upgrades.

ThayerCups server upgrade

We just upgraded the CUPS print server that handles print jobs from Mac and Linux clients. There should not be any change in the way you print from these platforms.  The new server is now easier to back up.

Application Virtualization

We are testing, “Application Virtualization”, a new technique for encapsulating applications to ease deployment to lab computers and client computers.

Atrium Help Desk continues

We continue to hold our “Atrium Help Desk” from 3:00-4:00 on Tuesdays and Thursdays. Come by the atrium if you have any computing-related questions.

In order to perform this upgrade, ThayerFS will be taken offline and will be unavailable from 8:00am until 2:00pm on December 15th. Our hope is that the upgrade will be completed in less time than this, but it may take the full six hours. Once the system is back online, we will send e-mail to the community. Note that this outage will not effect e-mail, internet, or any other network services other than ThayerFS.

During this upgrade, we will also be moving the data in the Research share to different physical disks. During this move, Research will be accessible, but will be read-only. This move could take up to ten hours from the time that ThayerFS is back online, but may be done sooner. We will send separate e-mail regarding the status of this move.

If you have any questions, please feel free to e-mail computing@thayer.dartmouth.edu.

Your Friendly Computing Services Team

AMANDA approaches backups by using native programs such as postfix, tar, dump, xfsdump, etc within a client/server relationship. The catalog database is store in flat tar.gz files instead of a more complicated (but perhaps more scalable) database such as MySQL. AMANDA’s claim to fame is its scheduler. You tell AMANDA you backup cycle (say 30 days), the set of clients/directories/files you want to backup, and she takes it from there, automatically balancing backups (both full and incremental) throughout that time period. While this is great for load balancing, as I understand it, the administrator is only guaranteed that backups will happen during the cycle, not exactly when those backups will happen. AMANDA is also very fast. In my testing (two Dell PE2850 machines running Ubuntu over Gig-E networking with an IBM LTO-3 tape drive) the backup averaged ~50 MB/s between the client (to the holding disk) and then ~77 MB/s from holding disk to tape. Where AMANDA shows her age is dealing with tape libraries. The only way I have found to handle a tape library with more than one drive (ours has four) is to create separate AMANDA configurations, each with its own tape drive and set of slots in the library. This negates much of the smart scheduling AMANDA provides, as you have to manually balance which directories/file to backup in each AMANDA configuration. Another gotcha with AMANDA is the inability to append data to tapes. Overall, AMANDA is simple and fast, but makes coping with a large tape library somewhat cumbersome.

Bacula is a holistic backup solution. Everything needed to use Bacula is included with Bacula. Bacula is implemented with a client/server relationship, using its own protocols and your choice of MySQL, PostgeSQL, or SQLite to store the catalog in. Scheduling is done in the standard fashion of telling the program what days/weeks/months you want fulls and incrementals to occur. Bacula handles tape libraries very well, allowing multiple tape drives to co-exist in a single configuration. Performance is adequate. Using the same hardware and OS as in the AMANDA tests. The same backup (~90 GB) that was used in the AMANDA tests averaged ~30 MB/s from client to holding disk, and then ~63 MB/s from holding disk to tape. Overall Bacula provides a slick program, great documentation and support for advanced features, at the price of some speed.

In conclusion, AMANDA and Bacula are both excellent programs, with different strengths. From my research, the difference in network speed between AMANDA and Bacula is due to Bacula pumping everything through one TCP port, while AMANDA fires up several dumpers, using many TCP ports to transfer the data. When backup up more than one client, the aggregate speed Bacula provides increases.