Subscribe to the Free Print Edition!
Celebrating 25 Years

GCN Home > 11/22/04 issue

Which test is best?

By Joab Jackson, GCN Staff

Popular benchmarks can be misleading; experts say the best way to evaluate systems is still to set your own metrics

Last month, NASA announced it was running what may well be world’s fastest computer. The Columbia, named in honor the crew of the space shuttle Columbia, consists of 20 interconnected SGI 512-processor systems. NASA said the computer reached a sustained peak rate of 42.7 trillion floating-point operations per second.

The announcement followed a declaration last month from IBM Corp. that it had the world’s fastest supercomputer, one it plans to deliver to Lawrence Livermore National Laboratories next year. IBM claimed that its system achieved a sustained performance of 36.01 TFLOPS in the IBM laboratory.

IBM’s number had edged out the performance of the Earth Simulator in Yokohama, Japan, which had held the title of the fastest computer of the last few years. That unit, built by NEC Corp., can run at 35.86 TFLOPS, according to the simulator manager’s submission to the most recent Top500.Org list of the 500 fastest supercomputers.

All these organizations used the same benchmark to test their computers, one called Linpack. One of the dirty little secrets of the government high-performance computing world, though, is that Linpack doesn’t measure anything useful, other than how quickly a supercomputer can execute the Linpack test.

Linpack itself is a collection of subroutines that solve linear equations. In the scientific community, it has long been discarded in favor of the Linear Algebra Package, or LaPack.

Grading the tests

The focus on Linpak illustrates one of the problems of evaluating technology: Widely accepted performance measures—by which products or systems are rated—may not really apply to what you need. At the PC level, CPU clock speed has been the most common gauge of performance, but other factors—such as CPU architecture—could actually be more important. At the supercomputer level, benchmarks have been the common denominator, but some experts doubt their viability.

“We really don’t care about Linpack that much,” said Walt Brooks, chief of NASA’s Advanced Supercomputing Division. “We’ll measure it because everyone wants us to measure it, but we’re much more interested in reliability, usability, productivity and what really happens with the system.” Columbia, for instance, will do such large jobs as hurricane forecasting, supernova simulation and next-generation aircraft designs.

More news on related topics: Enterprise Architecture, Executive Center, IT Infrastructure, Management



GCN Popup