Big data is just a stepping stone
Connecting state and local government leaders
Government IT managers need to shift the focus from the data producer to the data consumer and zero in on information extraction.
Recent articles about Pandora’s and Netflix’s use of big data illustrate why government IT managers should not just focus on data management, data collection and even big data processing. They need to shift the focus from the data producer to the data consumer.
A Wall Street Journal article on Pandora describes how it is mining its big data troves to make inferences on its users’ political affiliation, demographic groups, parental status and income. It is those inferences that are of value to the music service’s advertisers, not the music data listened to.
Meanwhile, the Atlantic looked at Netflix’s efforts to extract useful information from its big data. Netflix’s creation of micro-genres (a form of taxonomic categorization) like “critically acclaimed emotional underdog movies” improves the chances consumers will discover movies that match their tastes and experience based upon past viewing habits.
In both these cases, we see big data is the stepping stone for consumer-centric information production. The Netflix micro-genres are not the trove of big data on movie viewing, or the movie data itself. Instead, it is useful information mined from that data. Likewise, the data containing Pandora users’ demographics and preferences create a way for advertisers to target buyers.
In The Semantic Web, my co-authors and I described and demonstrated how an inference rule allows you to derive conclusions from a set of premises. It is important to understand that those inferences are not the data; they are useful information derived from the data. Thus the big data is the stepping stone necessary to derive useful information for consumers (in this case, Pandora’s advertisers). In other words, in order to make use of big data, we need to shift the focus from the producer to the consumer.
This shift can be explained in terms of push-driven versus pull-driven processes:
“We must replace this uninformed data collection with engaging with business managers and line employees to learn what information they need to be effectively productive. After information requirements are clear, we pull the information production process by pulling and assembling data from multiple data sources. This is the essence of consumer-driven information production. Like the Toyota production process, it is a ‘pull-driven’ process.” – from Information As Product.
This same phenomenon of brute-force data collection is likely to happen again with the hype and rush to big data. Big data for big data’s sake is a fool’s errand. Instead, a big data effort should be initiated by a consumer question.
A good example is President Obama’s re-election campaign of 2012. While the big data case study I did in the Great Cloud Migration is fairly long, the description of a fine-grained voter model is particularly relevant to this article: “This fine-grained model of each voter included four scores: the likelihood to vote, the likelihood to be an Obama supporter, likelihood of being an Obama donor (or volunteer) and the likelihood of being persuaded by a particular topic. Each of these scores was measured and modified after each interaction with the candidate, and that was how the campaign would micro-target voters for promotions.” The need to answer these questions drove the big data collection. In other words, the cart is big data and the horse is the consumer (or in this case, the voter).
One final note of theoretical interest is that Pandora’s music “genome” of songs and artists and Netflix’s movie “genome” of movies and actors/actresses are pure metadata (the music and movies being the data). Given that, the inferences on that music genome and the micro-genres of that movie genome are “meta-metadata.” Given the proven value of this metadata and now meta-metadata, every organization that does not leverage a metadata catalog should re-examine the issue.
I recently reviewed a large government organization’s “metadata manifesto” and was pleased to learn IT managers felt so strongly about the issue that it warranted the boldness of a manifesto. Does your organization have a metadata manifesto? And in regards to big data, does your organization see big data as an end in itself or as a stepping stone?
Michael C. Daconta (mdaconta@incadencecorp.com or @mdaconta) is the Vice President of Advanced Technology at InCadence Strategic Solutions and the former Metadata Program Manager for the Homeland Security Department. His new book is entitled, The Great Cloud Migration: Your Roadmap to Cloud Computing, Big Data and Linked Data.
NEXT STORY: What is virtual network computing?