Yottabyte Data Warehousing!

… and you thought you had information overload problems!!  The tools required to store this level of data are going to be something to be seen, and the mind boggles at the thought of how and what you use to make sense of all this data.

Yottabyte – A yottabyte is 2 to the 80th power, or 1,208,925,819,614,629,174,706,176 bytes.

Military Grapples With Information Overload

Surging surveillance data threatens to overwhelm the military’s ability to deal with the information. A report from a defense advisory group is calling for new data analysis technology and for taking a cue from Google.

By Thomas Claburn

InformationWeek

Information overload has become a significant challenge for the U.S. military and will require new analysis software and a Google (NSDQ: GOOG)-style cloud infrastructure to manage massive data sets, a U.S. defense advisory group report finds.

The December 2008 report, “Data Analysis Challenges,” was initially withheld from the public. It was obtained by the Federation of American Scientists’ Project on Government Secrecy through a Freedom of Information Act request.

The report, written by JASON, a group that provides advice to the Department of Defense (DoD) through the non-profit MITRE Corporation, says that the massive amount of sensor and imagery data being gathered is becoming increasingly difficult to store, analyze, and integrate into defense systems.

For example, a DoD surveillance system called Constant Hawk typically produces 10’s to 100’s of Terabytes of data over a period of a few hours. For that information to be useful, it has to be stored, analyzed, and distributed quickly.

The report, however, cites concerns voiced by members of the defense and intelligence communities that much of the surveillance data gathered isn’t made useful.

“Seventy percent of the data we collect is falling on the floor,” MIT defense research scientist Pete Rustan said, according to the report.

And the problem is likely to get worse. “As the sensors associated with the various surveillance missions improve, the data volumes are increasing with a projection that sensor data volume could potentially increase to the level of Yottabytes (10^24 Bytes) by 2015,” the report says.

Jonathan B. Spira, CEO and chief analyst at research consultancy Basex, author of the forthcoming book Overload!, and organizer of Information Overload Awareness Day on August 12, says information overload is a real problem in the workplace, in government and in the military.

“We’ve seen on the military side, many instances where information overload can create a whole new kind of fog [of war],” he said.

Information overload costs the U.S. economy $900 billion per year, according to Basex.

The JASON report discounts some of the more extreme projections about data volume growth and recommends that the DoD deploy infrastructure similar to that used by Google, Microsoft, and Yahoo. It also sees military applications for the Hive language used by Facebook for data warehousing.

The major problem the DoD faces will be in the area of automated information analysis. “The notion of fully automated analysis is today at best a distant reality, and for this reason, it is critical to invest in research to promote algorithmic advances,” the report says. “One way to effectively engage the relevant research communities is through the use of grand challenges in the area of data analysis.”

Spira sees information overload as a broader problem, one that won’t vanish with the development of improved automated information analysis technology. He described a cybersecurity conference at a Maxwell Airforce Base, where military brass had gathered to discuss cyber threats. Emerging from the talk, the generals found they had no e-mail, he said. It turned out that the base’s e-mail system had been taken down, not by a cyber attack, but by an e-mail about a card game that got forwarded and, through too many reply-alls, multiplied until over a million messages overloaded the e-mail servers.

“We need to address a lot of different aspects of data and information overload, not just things that sound sexy,” said Spira.

Advertisements

About Andy Painter

A passionate Information and Data Architect with experience of the financial services industry, Andy’s background spans pharmaceuticals, publishing, e-commerce, retail banking and insurance, but always with a focus on data. One of Andy’s principle philosophies is that data is a key business asset.
This entry was posted in Business Intelligence, Information Management, Intelligent Enterprise and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s