The Bytes Behind Biology

BigBen's 21 cabinets hold 4,136 processors

A three-dimensional image of a human neurosynaptic membrane spins in the middle of a darkened room, its serpentine powder-blue profile passing only meters in front of my face. Magenta dots dance around the membrane; some bind to receptors attached to the tissue, while others vibrate wildly. The only sound is the humming of two projectors. Joel Stiles, director of the National Resource for Biomedical Supercomputing (NRBSC) at the Pittsburgh Supercomputing Center (PSC), begins to narrate, explaining that the magenta dots are calcium ions flowing into the membrane. Their binding triggers neuromuscular firing.

The image looks real enough to touch, thanks to two projectors and data rates 20-25 times higher than a standard DVD movie. "This transforms stereo display from something that's just kind of a gimmick to something that's extremely useful," says Stiles from the back of the room, the colors reflecting off the lenses of his 3-D glasses. As the camera zooms out, the shadow of the translucent gray nerve comes into view, showing the entire system. "This is what you hope it looks like in the normal case," he says.

BigBen's networking cables and switches

ﾩ Mark Bolster Photography

These movies, based on two software programs that he developed, are Stiles' tools for uncovering the action of neurosynaptic firing. Each model requires detailed programming and hundreds of hours of computing time. None of it would be possible without the processing power of PSC's supercomputers, which run through all the potential ways the neurosynaptic system might respond to an influx of calcium ions - action we otherwise couldn't see in real time. Like all the other projects associated with PSC, it would take years to do what the biggest computer, aptly dubbed BigBen, can do in a tiny fraction of the time. BigBen's 4,136 processors can accomplish in weeks the tasks that would require five to 10 years for today's fastest desktop computers.

That kind of power comes at a price. In September 2004, the National Science Foundation gave PSC $9.7 million to build BigBen. Overall, PSC received $52 million to build TeraGrid, which links nine computing centers (including BigBen), bringing together 280 teraflops of computing power. This network has produced an estimated 600-plus papers in 200 journals. And at PSC, a voluntary database of papers submitted by supercomputer users shows 2,200 papers published over the last five years.

Researchers have harnessed this computing power to produce the first atomic-level look at how certain proteins can bind and pass through the nuclear pore complex, thus confirming three of the four binding sites suggested by electron microscopy, and identifying six novel sites. Those findings resulted from using just one supercomputer for 16 24-hour simulations. Another project has corrected previous estimates of the number of binding sites on calcium receptors in the synaptic membrane. Now, BigBen is steadily churning through simulations of the 2.7 million atoms in action in the ribosome.

Lush greenery fills the wide windows outside most PSC offices, part of a squat brick building 14 miles from BigBen, and tucked among the outdoor restaurants and novelty shops of the Oakland neighborhood. The wide corridors of PSC are lined with multicolored posters depicting enzymes and mouse heart cells. Structural biology posters present ribbons of rainbow-colored molecules. Indeed, the wall art along the quiet floors of PSC make up for what is lacking in outward exuberance: Most employees sit before two monitors or more, clicking away at their keyboards with little chatter, some with their feet up on the desk, shoes off.

Nearly a month after moving into a new corner office on the fourth floor, Stiles' walls are essentially blank. He attributes this to having too much work and no time to think about decorating. Nonetheless, if your business is visualization, a clean, white wall could be a welcome break.

Computer models that can predict what will happen in a particular system have been used for decades, such as in weather forecasting. Because models run on a set of infinite possible outcomes, however, the work depends on supercomputers to run through each of the potential outcomes and pinpoint those that are more likely to occur.

In 2005, using MCell (Monte Carlo Cell simulation) and DReAMM (Design, Render, and Animate MCell Models) - two programs that he wrote and has continued to develop for more than a decade - Stiles' group showed that previous research underestimated the number of binding sites on the calcium receptors in the synaptic membrane. ¹ Preexisting models, based on electron microscopy and experimental work, had suggested four calcium binding sites to facilitate neurotransmission. Stiles created a model of single calcium channels with stochastic opening and closing, which allowed him to watch the influx of calcium through single channels, trace each ion as it entered through each channel independently, and look at the vesicle it happened to bind to, if any.

By varying the number of binding sites and relating that to the molecular data, all while tracking the release of neurotransmitter, the group came up with a different estimate of the number of binding sites. "It turns out that we find that we need many more than four biding sites for calcium," Stiles says. Otherwise, "the system is so insensitive that nothing would ever happen."

Inside the computer, a cooling fan is more than a foot in diameter.

ﾩ Mark Bolster Photography

The model enables Stiles to change variables in the system, such as knocking out some calcium channels, which mimics what happens with certain neurodegenerative diseases. He can also stimulate the cell more than once in the simulation, releasing a stronger concentration of neurotransmitters and recreating both short-term and long-term plastic changes in the brain. "We're always driven by wanting to answer new scientific questions," Stiles says. "You have no choice in doing the software development alongside the application, and that is an enormous task."

One of the pitfalls of the PSC research comes from believing it too much. It's a powerful tool that produces picture-perfect images, and the scientists must constantly remind themselves that what they see is a representation, not reality. "In our job we forget that we're looking at models, and need verification," says Stiles' coworker Troy Wymore. Computer-based researchers, he adds, must be careful not to say of their work: "This is nature."

Indeed, glitches small and large pop up. Later in the day, Stiles hovers over a computer workstation for one of the NRBSC software developers, Jack Chang, who scoots on his rolling chair between two computer terminals. The program has hit a snag resulting from an upgrade in the operating system software, which has stalled the work.

"It's just one of those silly problems with the operating system," Stiles reassures. "It's not minor," returns Chang, peering into the screen. His bright yellow t-shirt reads: "Supercomputing is our middle name."

"I said silly, not minor." Stiles accepts these speed bumps as just part of the job. By the end of the day the glitch is resolved, and work proceeds.

Two doors down from Stiles' office, Wymore stares at simulations he's created to uncover the secrets of an elusive enzyme, aldehyde dehydrogenase. ALDH is one of the largest families of enzymes found in most living organisms; in humans, it is involved in breaking down ethanol, for example. Wymore and colleagues at PSC are using simulations to pick apart, atom by atom, how this enzyme operates at its active site. They have discovered an unusual mechanism of action, whereby the enzyme activates a proton transfer from its main chain. Wymore and colleagues demonstrated that without the proton transfer, the enzyme produces a dead-end product, one that is at the end of an energetically inert pathway.

MCell simulation of the neuromuscular synapse between a nerve cell (translucent blue) and underlying muscle cell (green). The nerve contains an array of specialized structures, called active zones, from which neurotransmitter molecules are released when the nerve fires.

To make this model, a single active zone geometry was created with computer-aided design software based on electron microscope measurements. Cyan spheres are synaptic vesicles. Calcium ions (yellow) enter through the transmembrane ion channels and can also bind to many intrinsic buffer sites (magenta).

Courtesty of Joel Stiles / PSC

Thanks to X-ray crystallography and sequencing experiments, Wymore knows that the simulations he is running were, at least in structure, an accurate representation of what's happening in vivo, and he feels confident that the model has shed light on a completely new mechanism. New research by Wymore's group, to be published this month in Biochemistry, suggests a link between this mechanism and two metabolic diseases, Sjogren-Larsson syndrome and type II hyperprolinemia. The researchers argue that both diseases are caused by inherited mutations, which prevent ALDH from completing the necessary proton transfer. Wymore is already exploring, using simulations, whether it's possible to insert a molecule into the enzyme active site, which will facilitate the proton transfer, potentially leading to a viable treatment.

In order to estimate the action of ALDH at its active site, each of the simulations needs to be computed on a portion of the processors that comprise the supercomputers, some simulations taking several weeks. BigBen's processor time is split among the 2,000 researchers who use it; some projects can demand up to one million processor hours at a time. Wymore's Biochemistry findings required 15,000 hours on 900 of BigBen's processors, and 50,000 hours on another Westinghouse computer, dubbed Jonas. The largest allocation of processor time - six million processor hours - was given to researchers in particle physics. In general, chemistry and molecular and cellular bioscience research comprises about 49% of supercomputer usage.


According to Ralph Roskies, codirector of PSC, one of the most significant findings to emerge from the PSC supercomputer came from work by Klaus Schulten at the University Illinois, Urbana-Champagne. The researchers took advantage of TeraGrid (as fast as 35,000 desktop computers) to produce detailed images of the nuclear pore complex, which was then a little-known set of proteins that regulates the traffic of other proteins passing through the nuclear membrane. The complex is large, with a mass of 125 megadaltons in vertebrates. Schulten's research focused on the importin-￢ transport receptor and how it interacts with the nucleoporins in the complex that are involved in transport selectivity.² Schulten's group confirmed three of the four binding sites of the FG-Nups protein to importin-￢, which were previously identified in electron microscopy experiments. They also identified six novel binding sites, one of which has already been verified by experimentation.

In a dimmed computer lab, Stiles is teaching 14 high-school students a basic example of how to program a biological simulation. The students sit at their monitors, facing a long projection screen. Stiles is guiding them on designing a program that will count the number of blue and red ligands in a molecule.

The students follow along with Stiles, as he demonstrates the process of programming a simulation using MCell and DReAMM. Both programs work in tandem to visualize cell physiology and processes, such as the mechanism of action of neurotransmission at neuromuscular junctions (see the video on a frog neuromuscular junction).

The large projection screen casts three identical images of what the screen of Stiles' laptop is showing. At the back of the classroom, one of Stiles' research assistants, Jacob Czech (called "the magician" by other PSC employees for his knack for fixing computer-related problems), is watching the students carefully, in case they should miss a step or become confused.

"Ok, first make a new directory," Stiles says to the group in the computer lab. As the first step in programming a simulation, the students need to make a file to feed directly to DReAMM, the graphic simulator, in order to count the number of blue and red ligands in a molecule. DReAMM will display the physical representation of that on the screen.

In the command box Stiles types:

mkdir
BPSI
Ls
Cd BPSI
Cd unbounded fusion

The students follow along, with seemingly little difficulty. Once their programs are set up they can run DReAMM to visualize the simulations that MCell creates. The software produces a cloud of red and blue dots that represent the ligands of the molecule suspended in three-dimensional space. In this simple example, the students are looking at a freeze-frame snapshot of the molecules. On the floor above this classroom, higher-end applications of the software are being used to animate multiple proteins and even entire complex systems, creating life-imitating simulations whose components move and vibrate. Here, the students are starting with more basic tasks: They rotate the molecule cluster in space, and then change the shape of the atoms from triangles to spheres.

One student blurts, "How do you change the colors?" Others in the room echo eagerly, "You can change the colors?" Their enthusiasm temporarily interrupts Stiles' lesson. "Alright," he sighs. "I'll show everyone how to change the colors."

The students' eagerness to play with colors demonstrates the fun aspect of Stiles' software, which he tries to make as visually stimulating as possible. Although he has a PhD in physiology and an MD, Stiles spends a lot of his time as a movie director, creating story boards in the lab with his colleagues, drafting "camera" moves for DReAMM to produce, debating the scenes and level of detail to include. For his high-school students, as well as for the colleagues to whom he presents his simulations at conferences, he knows that teaching is 20% content and 80% theater.

Nicholas Nystrom, director of strategic applications at PSC, acquires supercomputers for the center and keeps all programs running smoothly. Keeping the computer system on the cutting edge is a nonstop job. It's a competitive game, as NSF awards the big systems only every four years (smaller system grants are awarded more frequently). As of now, according to Top500 Computer Sites, a Web site that has been ranking the world's supercomputers since 1993, PSC's BigBen ranks a mere 46th on the list of the world's fastest computers (number one is the Department of Energy's IBM computer at Lawrence Livermore National Laboratory). Next month, the NSF will reveal whether it has approved PSC's track-one proposal, in which it asks for $200 million to acquire the hardware for a new computing system with one petaflop of processing power, which is equivalent to 1,000 teraflops or 1015 (10 quadrillion) floating point operations per second. If granted, this new computer would likely be the fastest in the world, says Nystrom.

Processing power doubles every 24 months (according to Moore's law), though experts have shortened that doubling time to 18 months. In other words, all the work that goes into setting up the hardware, networking it, and getting it running, is all for naught after 18 months. As a result, researchers can never rest on their laurels or become complacent with their systems; no matter how fast or powerful, they must always be thinking about the next, even more-powerful system they'll soon need.

A supercomputer is not like the new car that proverbially loses value as soon as it leaves the lot, says Nystrom; it takes some time before he deems his system obsolete. PSC's team is constantly modifying BigBen and its network speed. Even since its installation in 2004, BigBen has been updated with dual-core technology, doubling its processing speed. By the time BigBen's demise draws near (less than three years from now), the center will get a new system and be well ahead of the curve once again, Nystrom notes.

PSC's track-one bid is competing against proposals from three other computing centers; the details of each are kept tightly guarded so competitors don't give away technology secrets. Only one group will get the grant, so PSC has also applied for a smaller amount of track-two funding. Just who will get that track-one grant remains a mystery. "We've heard every possible combination of outcomes," Nystrom says. "We wish we knew." In the meantime, the supercomputer in the Westinghouse basement continues to tick away, its red and green lights flashing in the darkness.