Big Data and a Portfolio Approach

Given the sharp decline of the cost of data storage and the emergence of scalable tools to explore and mine this data, more and more data is becoming systemically accessible every day.  We are just at the beginning of applying the information available among the growing data sets around us.

Big data is, well, big.  It is new.  And the tools emerging to access and harvest the information it contains are also new. Therefore, getting to real, useful information when exploring big data is a difficult task.

Many of the applications of big data have to been big as well.  And complex.  This complexity of application on top of what is already a complex myriad of nascent tools makes for a very brittle system.

I've been thinking about this differently.  My take is that we need to focus engineering lift on the complex methods and tools to extract information from data, and streamline and simplify the application.  Simple applications are easier and faster to build.  Faster builds allow for a quicker return on effort.  Product designers should therefore focus on thin web apps that leverage these vast, complex datasets.  Think of your initial applications as prototypes for your big data system...

How can we use big data in small, focused ways to improve our lives?  What "little" things can be extracted from available datasets and applied quickly?  How can the burdens of complexity be pushed down the stack, to simplify the application, and lessen the investment required before reaping any value?

Oh, and one more thing - there is another benefit to pushing as much of the engineering and complexity to the data processing layer.  This also enables a portfolio approach, whereby tens, hundreds or even thousands of apps can be built on a single data stack.  Why use a shot gun or even a sniper, when you can use an army to mine for value...

UPDATE: I just started playing with a new app that munges this thinking, Osito.  (Good overview from The Verge here.)  Basically, it's a single iOS app that leverages the portfolio approach to provide lighter, thin alerts given your personal data.  The product focus is triggers based on user location.  Interesting play - we'll see if it works...