Phases of Data Intelligence

"Big Data" has been a thing for a few years now.  As with any new idea, the hype and promise of what it is can overshadow the effort required to actually deliver.  Big data is no exception.

Many focus on the promise of "predictive" intelligence without understanding the effort of basic data collection.  Others focus on dashboards and other tools without building the infrastructure needed to deliver those pretty pictures.

Through conversations with many, I see the following stages to data intelligence:

Stage 1*: Data Collection - many systems, processes, tools already through off a ton of data.  For most industries and applications, this data is often not stored, let alone organized for use. 

Stage 2: Data Infrastructure - for storage and organization, quality infrastructure is required.  This is where much of the foundational innovation has happened in the last 10 years or so, that has spurred idea of big data.  The idea of it being too costly or too difficult to store and organize vast amounts of data is no longer true.

Stage 3: Data Visualization and Interpretation - this is an area that some skip by either hubris or eagerness.  Hubris is when those not in the trenches believe they know the right path to extract intelligence from data, and build accordingly.  Eagerness manifests by going after predictive intelligence before knowing what data and information is available. 

Stage 4: Data Intelligence - this is the stage where real value is delivered.  The steps above are the plumbing to get you to this point of actually learning from the information gleaned from data. This is the stage where action is taken, given what is learned.

Predictive intelligence is an extension of data intelligence, whereby historical data is used to preemptively make decisions in the future.  As "cool" as it is to do, there is so much that can be learned and value uncovered by effectively developing intelligence from historical data.

*Given that there is so much data thrown by existing systems and processes, I take the data stream as a given.  This may not be true for some markets / industries, but it is fast becoming the norm that the data is there for the taking / analysis / employment...