Big Data and Causation

The new book on Big Data by Viktor Mayer-Schonberger and Ken Cukier is a good intro to the age of big data, with a slightly optimistic telling of its benefits and a fair estimation and treatment of its risks. One thing the book points to remains troubling to me, weeks after reading: "Most strikingly, society will need to shed some of its obsession for causality in exchange for simple correlations: Not knowing  why but only what. This overturns centuries of established practices and challenges our most basic understanding of how to make decisions and comprehend reality". Yes, it is a revolutionary approach but is the prediction correct? The point is certainly well taken when you examine, for example, how marketing works in an era of big data. Is the same thing going to be true for cutting edge cancer research?  Or has Mayer-Schonberger picked up on something else, that statistical analysis cannot demonstrate causation, and the more data we have (and the more time we spend crunching it), the less time we will have for causal analysis? However, less time for causal analysis should not mean abandoning it. I still find this concept troubling and struggle with giving up the notion of causation - isn't that what the entire scientific process is about? I suspect that M-S is at leats partially correct in his factual view of the rise of correlation, but must causation therefore be diminished? Can it? I'm just not sure.

Andrew Keen needs a "chill pill"

This blog is about limitations on technology - those imposed by law and other rules, and those imposed by human nature. We always welcome a dose of humanistic-oriented technoskepticism as an appropriate antidote to rampant and un-thought-through) techno-utopianism. But Andrew Keen needs, in my daughter's words, to take a "chill pill". His two most recent works, "Digital Vertigo" and "Cult of the Amateur" are overly alarmist, viewing the end of civilization as nigh due to the rise of the amateur and the fool.  Yes, it is true - traditional institutional gatekeepers of moderated "truth" - great newspapers, encyclopedias, etc. are suffering and having their lunch eaten by blogs, digital startups and Wikipedia.  Yes, that will have a significant deleterious effect on what we consider to be appropriate and acceptable as knowledge. Yet Keen misses several larger truths at work. First, present conditions never continue, and neither will these.  Some of the large media and knowledge companies have adapted to the age of "digital diarrhea" and some haven't; others will certainly take their place. But no need to fret (yet) about them - the major players are doing what they can to reinvent themselves, and the winners in the race to replace them may not be so bad themselves. Second, digital overload, whether from amateurs or professionals, is giving rise to a colossal need for curation. Admittedly, some of that curation is coming from big data algorithms, but some (much?) is from human curation. Count me as dubious that big data, standing alone, can curate properly, or even write a proper and insightful sentence. Full disclosure - my daughter is thinking of attending grad school for journalism. Everyone tells her she's crazy except me. I think that quality writing and quality editing will always have a place - perhaps not in the locations it heretofore had, perhaps to be appreciated in different ways - but a place nonetheless. So, "chill out" Mr. Keen - current trends accentuate the amateur and the stupid in life, but there is plenty of reason yet for hope.