Government Computer News
Too much for NSA to mine?
Storage, computing power biggest hurdles for spy agency
The commotion over allegations that the National Security Agency has been secretly compiling data on millions of telephone calls made by ordinary citizens raises an interesting question: With the technologies in place today, how well can NSA actually mine the information it gathers?
Digg
De.licio.us
Slashdot
Technorati
There is no public information on the computers and software NSA is using, and the spy agency isn’t about to discuss them. But there are companies whose products are used to handle very large databases, and the challenges NSA would face in cross-referencing the information and looking for connections are well known.
Calling on AT&T
According to published reports, AT&T Corp. provided access to “Daytona,” a database management technology that it uses to manage its call detail record (CDR) database. “As of September 2005, all of the CDR data managed by Daytona, when uncompressed, totaled more than 312 terabytes,” according to the Electronic Frontier Foundation, which filed a class action lawsuit against AT&T in January.
If this figure is accurate, NSA’s database could be about 900TB, assuming Verizon Communications Inc. of New York and BellSouth Corp. of Atlanta, the other companies alleged to have provided information to NSA, have CDR databases of similar size.
The technology needed for data mining can be broken roughly into three components: storage, computing power and analytical software.
Data mining—looking for patterns in all that information that could reveal terrorist “sleepers” within the United States, for instance—requires a computer to have real-time access to as much of the entire database as possible.
“Storage is a combination of online accessible storage and offline multimedia storage,” said Robert David Steele, chief executive officer of OSS.net and a former CIA employee, who now champions the use of open-source information for intelligence purposes.
“My impression—strictly a professional guess—is that at least 75 percent of what NSA ‘knows’ is … offline and not accessible. … You cannot do good pattern analysis, including historical comparisons, without massive online storage.”
SGI has been pioneering ways to increase the amount of RAM available to its computers. While desktop computers have, on average, 512M to 1G of RAM, SGI has configured systems with terabytes’ worth of active memory.
![1105 Government Information Group [justice]](/images/1105logo_website.gif "Government Computer News [justice]")
© 1996-2008 1105 Media, Inc. All Rights Reserved.
