Researchers in quest for vaccines

Cluster management software provides massive computer power, aiding the study of carbohydrates and proteins

The Complex Carbohydrate Research Center, at the University of Georgia was faced with a significant technical challenge to carry out its research developing simulation techniques to study carbohydrates and their interaction with proteins. The simulations demanded a massive amount of computing power, requiring a solution powerful enough to address this. A High Performance Cluster environment was the solution eventually decided upon, integrating low cost, ‘commodity servers’ through fully supported cluster management software. The outcome was dramatic with greatly improved time taken to run complex simulations and greater speed and computing power when simulating experiments, resulting in the faster development of vaccines.

The Complex Carbohydrate Research Center (CCRC) is based at the University of Georgia, and we are involved in researching and developing specific simulation techniques to study carbohydrates and their interaction with proteins. I head up the CCRC team, using this research to develop methods of creating vaccines to prevent bacterial and fungal infections.

To carry out these hugely complex simulations, the team required a complete high performance computing solution that would bring a vast amount of computing capacity with efficient networking. This was essential to deal with the large data structures that arose from complicated computational code.

Our experiments require the processing of massive amounts of data to deal with complicated computational codes and low latency. We needed a powerful solution with good cluster management software that would automate the time-consuming, but critical administration tasks. HP was one of only two vendors that offered a suitable solution as it offered better value with the right system management tools from Scali to provide the robust and reliable product we required.

Challenges facing CCRC vaccine research
As a research team, our challenge is to study the interaction of carbohydrate and protein in water, using molecular dynamic simulations. Simulations are performed using time-tested AMBER simulation software, augmented with GLYCAM, a set of parameters developed by my team for use with carbohydrates. The calculations generate an enormous amount of data. They save every position of every atom at every time step throughout the simulation. Typically that means 100,000 time steps in a single simulation, and 100,000 atoms, each with three dimensions. A simulation can produce 10 gigabytes or more of data.

It is always amazing to think about the number of calculations required to produce that data. In practical terms, to simulate 100 nanoseconds of real time - less than the blink of an eye - can take several months on a single CPU. And what you really want is a microsecond, which is ten times as long to simulate. So you need bigger and bigger systems to generate meaningful results in a reasonable time frame.

Equally important is the need for precision. Running a simulation for weeks at a time compounds even the smallest error, so the advantage of 64-bit processing becomes critical. It drastically reduces the truncation and rounding errors that even a 32-bit system would introduce into the calculations.

We were investing vast amounts of time and resources building and maintaining its existing clusters. As more complicated simulations were being carried out, the demand for computing power increased. It was therefore necessary for us to reassess this clustering solution. We decided to look at commodity cluster computing.

Initially, I needed to establish the team’s computing requirements. I brought together four discarded workstations at the university, linked them with Ethernet, and ran a baseline simulation. Although it worked, the solution was not powerful enough to run the simulations needed for the CCRC’s work, but it did encourage our move to find another solution.

The next stage was for the team to build a cluster of 16 PC nodes with dual Intel Xeon processors. The major problem we faced with this was maintenance. There was no cluster management software good enough, and components kept failing. I was spending more time monitoring faults than carrying out experiments.

It had become necessary to find a software management solution that would allow my team to improve the manageability, simplicity and reliability of the high performance computing clusters required to carry out our work.

In 2002, I applied for and received a grant from the National Institutes of Health. This allowed me to assess the market and benchmark various solutions, leading me to purchase two Linux-based HP commodity clusters running a combination of Scali Manage and Scali MPI Connect cluster management software.

Cluster management improves manageability to remove administrative burden of HPC Cluster
The two commodity clusters we chose had to serve different purposes. The first deployment is an HP Integrity rx 2600 Linux-based 8-node cluster with dual Intel Itanium 2 processors, Myrinet interconnects, running Scali Manage and Scali MPI Connect, built to execute extremely demanding quantum calculations required for molecular simulations. I think of this as the ‘capability’ cluster, as it was running a high volume of jobs and processing the vast amounts of data required for very complicated simulations.

The second deployment is an HP ProLiant DL360 Linux-based 64-node cluster with dual Intel Xeon processors, Gigabit Ethernet and Myrinet interconnects, running Scali Manage and Scali MPI Connect, to allow large molecular dynamic simulations to run in parallel. This ‘capacity’ cluster allows many jobs to be carried out simultaneously dramatically reducing experimental time.

The HP High Performance Compute Clusters and HP StorageWorks RAID storage arrays were the ideal choice as they provide the low latency and fast CPU performance needed to model complex reactions at the atomic level, providing new insights into metabolic processes.

A factor that is very important in research is ease of use when it comes to IT system administration and management. Our existing clustering management processes were draining our resources - what used to take us a week we can complete in a day using Scali software. It was essential to have cluster management software that would automate the time-consuming, but critical administration tasks that I was facing on a daily basis.

The two clusters depend on Scali Manage and Scali MPI software and have brought a significant amount of benefits to us and the management of our clusters.

Scali Manage allows me to monitor all nodes easily, through a single interface. This has dramatically reduced our time spent in fault finding and monitoring and has enhanced overall cluster reliability. This is where the real problem lay previously - finding and assessing faults in both hardware and software. I’m particularly keen on the feature that allows commands to be sent from one central CPU across all other nodes in the cluster, as it dramatically reduces my administrative time and greatly increases productivity across the team.

Other benefits include being able to monitor and update the state of all interconnects, including Myrinet, which was not possible in the past.  The Myrinet interconnect ensures the low latency that is required to keep up with the CPU speed.  This is very important to the success of the cluster, as high interconnect speeds are a significant benefit of adopting cluster architecture.

A model of the interaction between the protective human antibody
mAb 1B1 (blue) and the surface polysaccaride from the bacterium group B Streptococcus (red and yellow). The structure was generated using comparative modeling and refined with explicitly solvated molecular
dynamics simulations. It is consistent with all NMR and immunological
data and provides a structural guide for vaccine optimization.

The integration of Scali MPI Connect into the solution allows us to run various simulations much faster over the clustering interconnects, therefore allowing jobs to be completed more easily. This makes job queuing run more smoothly, allowing for quicker and more reliable data management.

The ease of use and quick installation that both Scali Manage and Scali MPI Connect brought to the cluster management meant that both clusters could be installed within a day, whereas this had previously taken weeks. It has freed up valuable time and resources that we can now spend on experiments further our research into vaccine design.

The grant that the team received has allowed us to implement a complete fully supported, reliable and cost effective solution. The greatest cost that is often attributed to clustering is the implementation and administration of ad-hoc software applications. Scali’s out-of-the-box software has enabled the CCRC to increase the efficiency of running its cluster solution while removing the headaches and administrative workload it has experienced in the past.  It is clear that in terms of initial outlay for the clusters, the significant benefits that have been achieved through their implementation in terms of productivity, reliability and manageability have brought a significant return on investment.

Commodity Clusters provide the key to vaccine simulation
You just can’t cut corners in the simulation, which is one reason a high-performance cluster is so critical to our work.

A significant amount of research has been done to come up with a single 3D structure for carbohydrates. But, at best, this might represent an average shape; and at worse, a shape that’s not relevant. Using a computational model lets the team study the time dependency of the shape of carbohydrates.

Scali’s fully supported and robust features integrated into the HP solution made it the ideal choice for my team and I to carry out our research. The benefits that the Scali software has brought to our overall cluster management, including quick installation and improved workload monitoring has been so extensive that we are planning to deploy it across other clusters in the laboratory.

So far, our work has enabled us to reasonably model the structure of carbohydrates. Now the challenge is accurately predicting the energy of the interaction. The next step is determining if drugs can inhibit an unfavourable interaction; and then, whether that information be used to develop a vaccine.

Drugs that are carbohydrate-related already exist to combat viruses like flu. With our research we want to develop to the next stage - to stimulate those interactions in the immune system and develop vaccines to stop these illnesses ever occurring. This would be an excellent breakthrough.

By Dr Robert J. Woods, Complex Carbohydrate Research Center, University of Georgia