LinkedIn and the New Economy

The following is either A) an insightful, educated observation of the true reality of the disruption underway of the middle class in the US, or B) a misguided diatribe that is a result of the availability heuristic and a "general" education.

For the purposes of this discussion, we will define the middle class as those that earn +/- 50% of the median household income within the US - currently ~$50K and change.  (This number has apparently been going down over the past decade or two.)

Over coffee with a friend yesterday morning, our conversation diverged to how our lives would be different had we had the tools and knowledge we have today even 10 years ago.  The world today is different, and the skills, tools and other resources necessary to thrive are different too.  My anecdote is that, had I had LinkedIn (and the attached awareness of the value of a nurtured network) while I was traveling the world on behalf of President Clinton, I would be in a different economic rung than I am now.   At the least, "change" would be "easier."

Don't get me wrong - my wife and I live well.  We are in the top 20% in terms of income, likely the top 10%.  I was born in at best the second 20%, and I grew up as our economic situation continued to

improve.  By the time I entered college, my family was well ensconced int he top 20%.  However, my take on the catalyst that enabled this rise within the socio-economic strata is that my Dad decided to make a change in 1980.  He switched careers, from being a school psychologist to pharmaceuticals.  And, with this change cam great opportunity, which he seized.

I remain in contact with many of my White House colleagues, with many reconnections made through my LinkedIn and Facebook accounts.  My point is that I cannot remember let alone contact people even half the people I worked with back in those days, people that likely could be helpful to me now and in the future.  Change is easy for me now, but it could be a lot easier with a wider, more diverse network.

My hypothesis is that the supposed "shrinking middle class" is less a result of a perilous economic attack, and more a result of the disruption of how to succeed - not unlike what has happened to music, is happening to journalism and television, and will soon happen to higher education, among other industries.  Lost in this disruption is a middle class that has a job for life, is promoted every 4 - 5 years, does roughly the same job for most of his / her career.  In it's place is an agile, undulating timeline of new roles, new responsibilities, new companies, and new colleagues.  There is no straight line in one's career trajectory, and those that understand that and equip themselves for that fact, prosper.  Those who do not stagnate.  Those that have built a network for change thrive.  Those that do not, wither.

What we are seeing in the census data is this shift.  The top 20% consists of those that understand and have adapted to this disruption, and are reaping the reward, hence the continued growth of their share of income.  As more understand this new world, more will prosper.  As our system adapts to the new reality, so too will the distribution of income.

Or not.  We shall see...

China

I had the fortunate opportunity to travel to China for the first time last week - what an amazing place.  I got on the plane alone, with no colleagues, and only an itinerary of my flight there and my flight home.  I was invited as a guest of a Chinese entrepreneur.  I cannot imagine a better way to see China than as a guest of a business leader.  And, being alone enabled me to consume the experience at all times - no opportunities to fall in to catty "American-centric" conversations.

I come away from the trip seeing more similarities than differences - among the people, within business, and even in how this Communist government operates.  Conversations were frank, direct, and untethered by political doctrine or even perceived social norms of Chinese culture (unlike my experience in Singapore).  The mode of business is one of service and immediate opportunity.  And, the government is trying to have the best interests of its people at heart, despite its often authoritarian ways.

I have a newfound sense of scale - there are 10 cities there larger than New York.  As I explored Dali City, a city roughly the size of Austin, my stomach churned at the site of what appeared as overbuilding to me at the time.  The bulk of the city appears to have been built int he 80s, if not prior.  It consists of Russian-style architecture, or Bali style native to the region.  Yet, there were several complexes recently completed, and by my count well over 50 projects underway - everything from a 5-star hotel to 5 - 10 building complexes.  It appeared that they are expecting the population to double overnight.

However, after discussing it with my Chinese friends, I realized how little is needed to fill these new apartment buildings, given the scale of growth underway and the vast population still living in the rural countryside.  I am still trying to understand how this scale impacts my view...

Hoffman Misses Key Asset of Current Diploma System

Reid Hoffman pens an interesting piece on how the diploma needs an upgrade.  Though I agree wholeheartedly with his premise, he misses one critical point that enables the current system to thrive - the value of fuzziness.

The root of any economic system is information - who has it and how good is it.  The value of a diploma from a 4-year institution is no different.  Hoffman describes well the pains some people have in clearly articulating the value of their education beyond the blunt instrument that is the diploma (the "sell" side of diploma value), and the tactics employers use to reduce their pools to the most likely candidates (the "buy" side).

However, he neglects the fact that this system provides access that otherwise would not be available for many, given the fuzzy reality of a given person's credentials due to this bluntness.  Think of the kid that skated by without going to class, or the one who only took the "easy" courses.  Would they have had the same opportunities if the system of measurement had been more fine-grained?  Will they engage in a system that penalizes them?

There is another similar issue with fuzziness that will stifle the embrace of a more modular system by the employer  - the fact that most employers do not have the information they need to hire well for a given role.  There is so much bias and prejudice that clouds the current hiring process in most firms that more information could hinder an already clunky process, rather than help. More information generated by a fine-grained system will take more effort from the hiring manager to sift and understand as well.  Roles will have to be broken down in more detail, and more time may be required to assess potential fits.

A critical key to any system that attempts to improve an imperfect but functioning system is to ensure these fuzziness issues are addressed on both the buy and sell sides.  Failure to address them adequately will stop any potential replacement of an archaic yet functioning economic system.

Data in Context

Garance Frank-Ruta clarifies the context surrounding the data "discovered" that show former IRS Commissioner Douglas Shulman visiting The White House 157 times during his tenure.  The short story is that the data used to make that claim is imperfect.  A large majority of the supposed visits were in fact unfulfilled invites.  Another claim made from the same dataset is that he visited more often than any cabinet member - another falsehood given that the system referenced is used primarily to allow access to those walking in to the complex - cabinet members, given their seniority, are able to drive on to the White House complex.

This is yet another example of how important it is to leverage data thoughtfully.  Intelligence requires deliberate thought, not quick assertions and grandiose conclusions.  Minimal effort would have reveled the imperfections of the data referenced.  (The system used was built to track appointments within the White House complex, but only for meetings and "typical events.  Access lists for larger events often forgo the use of this system, as do appointments involving more senior government officials cleared to drive in to the complex.)

To ensure one does not fall in to this trap, there are three questions you must first answer, before acting on the information gleaned from a particular dataset:

1. How was the data collected?
2. What specific data is included in the dataset?
3. And, most importantly, what specific data is NOT included in the dataset?

Only with such context can you begin to understand the information available...

Global GDP

Let this one marinade for a minute:

Information Efficiency vs. the Boogyman

I hate when writers use the boogyman to scare people.  Michael Carney does just that with his article on personal data, "You Are Your Data: the Scary Future of the Quantified Self Movement".

I don't negate the fact that a small minority will "do evil" with the growing exposure of personal data.  My point is that someone of Michael's stature and position should not focus on what will undoubtedly be a small faction, at the expense of the larger, more bountiful majority.  The quantified self (and an exponentially increasing other sets of data) are and will continue to deliver value, much of which we are only beginning to see.

From Michael,

For those of us who don’t measure up compared to the rest of the population, the outcome won’t be pretty.

But what about those that are unnecessarily penalized, given today's information inefficiencies?  The truth is that the industries he cites become more efficient with more (personal) data.  Insurance is at it's heart based on information - the more information available, the more effectively and efficiently risk can be priced.  The more risky clients pay more.  Market dynamics at work.

Health insurance, even home mortgages, are quantified bets given the information made available.   Yes, people will have to pay more, but others will have to pay less.

He finishes with an acknowledgement that he is not focused on the value.  Rather, he bases his argument on the need for user awareness.  I agree that privacy policies and terms of service documents need more transparency and less legalese. Using the boogyman to make the point is wrong.

Calling Bullshit on Big Data

This article has a decent list of ways to call bullshit on data-driven analyses.  Click the link for context, but here are the top points:

1. Focus on how robust a finding is, meaning that different ways of looking at the evidence point to the same conclusion. 
2. Data mavens often make a big deal of their results being statistically significant, which is a statement that it’s unlikely their findings simply reflect chance. Don’t confuse this with something actually mattering. 
3. Be wary of scholars using high-powered statistical techniques as a bludgeon to silence critics who are not specialists. 
4. Don’t fall into the trap of thinking about an empirical finding as “right” or “wrong.” 
5. Don’t mistake correlation for causation. 
6. Always ask “so what?” 

As often occurs with an emerging technology theme, the glitz and glam of the shiny new thing that is big data often overshadows the real value.  The above list is a great start in being sure that the data product or opportunity being pitched truly can add value to your mission.  

#3 is an interesting one - I see a trend in the emerging big data space that vendors and others seeking to exploit big data too often move to high end, overly complex mathematics, when more basic, easier to understand models would suffice.  This is especially true when building out new applications on top of large datasets.  You will often get to the productive answer faster by building simple prototypes before investing more expensive resources.  Data modeling is no different.

#5 above is a particularly important point.  My sense is that it is difficult for most to logically separate the concepts of correlation and causation.  I find myself jumping too far too often, by inferring to much import on a basic correlation that lacks any evidence of causation.  

At the end of the day, high end mathematics do not negate basic economic theory.  Be smart - don't forget your whits when digging in to big data...

Munging Moore's Law and Gay Rights

Moore's law states that computer processing power will double every 18 months or so.  There has been all sorts of extrapolations as to what this may mean to us as a society, the Singularity being one.  I've got another: Gay Rights.

I did a paper back in college (late 90s) on gay marriage - I still remember the feeling of astonishment that, by that time, no state had yet allowed same-sex couples to marry.  If I recall correctly, only a few allowed civil unions.  As of this writing, 9 states now allow same-sex marriages, and several others are well on their way.  That is a major cultural pivot in just 15 years.

My take is that the speed of the pivot has a lot to do with Moore's Law, or rather, the infrastructure it has enabled.  As computer processing has grown exponentially, so too has the speed of communication.  We have moved from The Pony Express to the Daily Paper to the 24-hour News Cycle to now near instant delivery, with each leap coming faster than the last.  In a similar vein, social networking has expedited the sharing of opinions and thoughts among friends.  What used to happen periodically on the front porch is now a constant stream.  Communication is exponentially faster, and so too are its persuasive properties.

As opinions change, the impact of that change radiates with rapid speed.  As one friend openly seeks to understand marriage equality, all connected friends are exposed to this shift.  Even as a lone NBA player comes out as being gay, the rapid dissemination (and exploration) of this story takes over like never before.  As with Moore's Law, change is happening exponentially faster.

David Brooks, Your Premise is Off!

I've already blogged about some of David Brooks' writing on big data.  Though it is admirable that he is taking the time to delve in to the emerging world of data, he needs to apply some differential thinking to the information he is collecting.  In this piece, his premise is again off:

The theory of big data is to have no theory, at least about human nature. You just gather huge amounts of information, observe the patterns and estimate probabilities about how people will act in the future.        

This is not the theory of big data - this is a small sliver of what is and can be done with the explosion of structured data that is popping around us.  To diminish the power of big data to just what can be gleaned through "estimated probabilities" is to focus on the tree and not the forest.

The power of data is in the information it contains, not the method by which it is extracted.  And the limit is our imagination.

The Philosophy of Data

In this article titled, "The Philosophy of Data", auther David Brooks asks:

What kinds of events are predictable using statistical analysis and what sorts of events are not?  

Now, I know an editor likely created the title, but his article limits the value of data to insights derived from statistical analysis  - as if that is the only means to extract information from (big) data.

I think this is the wrong question to ask.  This may be a bit optimistic, but my belief is that data analyses can answer most any question.  The problem (and opportunity) lies in ensuring the data contains the necessary information to answer the question - a problem we have only begun to explore.

In the same article:

...we tend to get carried away in our desire to reduce everything to the quantifiable.

Data is not just about quantification; it's about information.  We are only at the beginning of collecting, structuring, and even analyzing data.  My belief is that we will see great advances in this processing, which will in turn unlock new possibilities for data-driven insights.  Such innovation will enable analyses and insights never before possible.  Data will inform questions we don't even yet know to ask.

Big Data and Hiring

I came across this interesting delineation of the reasons for Ron Johnson's failure at JC Penney.  Hiring is another interesting bastion of opportunity to leverage data for improvement...

I spent some time at PeopleAnswers in it's early days (I was employ #3!) - a business that has scaled behavioral testing to improve hiring and recruitment.  They have (very effectively) attacked part of the problem - exposing our innate selves that drive behavior to potential hiring managers.  This innateness is the foundation of our potential succes. But it is not our whole selves - our experience, our passions, and other variables also play a role in determining our career success.

I wonder what a systemic understanding of JC Penney, Target and Apple's characteristics, culture, products, etc., might have told the JC Penney board, when coupled with Ron Johnson's behavioral profile and experience?  Might they have seen the mismatch sooner?

Big Data and a Portfolio Approach

Given the sharp decline of the cost of data storage and the emergence of scalable tools to explore and mine this data, more and more data is becoming systemically accessible every day.  We are just at the beginning of applying the information available among the growing data sets around us.

Big data is, well, big.  It is new.  And the tools emerging to access and harvest the information it contains are also new. Therefore, getting to real, useful information when exploring big data is a difficult task.

Many of the applications of big data have to been big as well.  And complex.  This complexity of application on top of what is already a complex myriad of nascent tools makes for a very brittle system.

I've been thinking about this differently.  My take is that we need to focus engineering lift on the complex methods and tools to extract information from data, and streamline and simplify the application.  Simple applications are easier and faster to build.  Faster builds allow for a quicker return on effort.  Product designers should therefore focus on thin web apps that leverage these vast, complex datasets.  Think of your initial applications as prototypes for your big data system...

How can we use big data in small, focused ways to improve our lives?  What "little" things can be extracted from available datasets and applied quickly?  How can the burdens of complexity be pushed down the stack, to simplify the application, and lessen the investment required before reaping any value?

Oh, and one more thing - there is another benefit to pushing as much of the engineering and complexity to the data processing layer.  This also enables a portfolio approach, whereby tens, hundreds or even thousands of apps can be built on a single data stack.  Why use a shot gun or even a sniper, when you can use an army to mine for value...

UPDATE: I just started playing with a new app that munges this thinking, Osito.  (Good overview from The Verge here.)  Basically, it's a single iOS app that leverages the portfolio approach to provide lighter, thin alerts given your personal data.  The product focus is triggers based on user location.  Interesting play - we'll see if it works...

Data Science vs. Data Intelligence

Sean Gourley gave a very interesting talk at GigaOm's Structure Data conference last month.  I have repeated his ideas around data science vs. data intelligence in several conversations. (Stacey Higginbotham does a great job distilling the talk here - the full talk is embedded below.)

He lays out the idea in one simple chart:

He also provides a few rules of the road about data:

• Data needs to be designed for human interaction
• Understand limits of human processing
• Data is messy, incomplete, and biased
• Data needs theory
• Data needs stories...  Stories need data

I have seen first hand bubble-like aspirations for what "big data" plus "data science" can offer.  Because the technologies are new, and so many are now becoming aware of the power of high-end statistics and machine learning, data science is perceived to be larger than it actually is for may.  

It is a tool, a method to solve problems big and small.  It isn't an answer.

Dusting it off...

After a multi-year hiatus, I am dusting off the blog. Since my last note, my company (Nico Networks) was purchased by The Washington Post Company. My partner and I joined what became WaPo Labs.  This move afforded me the opportunity to operationalize my thoughts and ideas faster than I could document them (how's that for an excuse?).

Through this experience, we were able to iterate and expand upon many of the ideas discussed to date in this blog.  However, instead of being limited to political campaigning, we were afforded the resources of one of the largest media organizations in the world.  As opportunity expanded, so too did the ideas...

Reflecting back, a common thread throughout has been the application of information culled from data.  The earliest applications we did on behalf of our largest client, Catalist, were built upon their voter file data.  At Labs, the team continues to expand and iterate on the application of information extracted from past activity streams, and from deep text analytics happening on the company's vast corpus of content built over decades.  We accomplished a great deal, and I am excited to see what continues to come from what is an amazing and very talented group of innovators.

We are only at the beginning of what has already been coined as the information economy.  The technology and expertise to efficiently extract usable information from the growing data sets that are emerging around the world is just being discovered.  As more data becomes structured, more interesting and never-before-seen deductions and associations can be made.   As new technologies and capabilities are applied, new stories can be told.

The brave new world is emerging, and I am excited to be a part of it, to continue to explore how information from data can change the world.  More to come...