Skip to main content

Truth And Hypothesis

A July 2008 Wired cover story titled The End of Science, resonates with my own experience with data and analytics - specially in light of the battles that one has to wage with those who are convinced that there a priori needs to be a statistical model to proceed with analytics. Apparently, analysis means nothing without a hypothesis. I believe with the right tools, the data should speak for itself - the models are not necessary.

This is however a hard case to make for someone (like myself) who is approaching the problem from a common-sense and industry experience standpoint without a string of accreditations and doctorates appended to their name to lend gravitas to their perspective. Folks like us don't earn the respect of the pure math nerds very easily. They tend to view us as slackers who want to bypass the rigors of the scientific method. So we soldier on, hobbled by models and hypothesis that are not even directionally consistent with what the data says about itself.

It was heartening to see this idea echoed by Google's research director Peter Norvig who says of statistical models "All models are wrong, and increasingly you can succeed without them". The skills and talents of statisticians are much better served in finding the dominant patterns in sets of correlated data. They can help whittle the field from 100s of X's that result in a Y to a handful of the most high-value ones. Forcing a model on data is almost detrimental in that you can spend the bulk of your time disproving the hypothesis and search for one that will stick. What is more, you could miss out on the real story the data has to tell.

Comments

Popular posts from this blog

Part Liberated Woman

An expat desi friend and I were discussing what it means to return to India when you have cobbled together a life in a foreign country no matter how flawed and imperfect. We have both spent over a decade outside India and have kids who were born abroad and have spent very little time back home. Returning "home" is something a lot of new immigrants like L and myself think about. We want very much for that to be an option because a full assimilation into our country of domicile is likely never going to happen. L has visited India more often than I have and has a much better pulse on what's going on there. For me the strongest drag force working against my desire to return home is my experience of life as a woman in India. I neither want to live that suffocatingly sheltered existence myself nor subject J to it. The freedom, independence and safety I have had in here in suburban America was not even something I knew I could expect to have in India. I never knew what it felt t...

Under Advisement

Recently a desi dude who is more acquaintance less friend called to check in on me. Those who have read this blog before might know that such calls tend to make me anxious. Depending on how far back we go, there are sets of FAQs that I brace myself to answer. The trick is to be sufficiently evasive without being downright offensive - a fine balancing act given the provocative nature of questions involved. I look at these calls as opportunities for building patience and tolerance both of which I seriously lack. Basically, they are very desirous of finding out how I am doing in my personal and professional life to be sure that they have me correctly categorized and filed for future reference. The major buckets appear to be loser, struggling, average, arrived, superstar and uncategorizable. My goal needless to say, is to be in the last bucket - the unknown, unquantifiable and therefore uninteresting entity. Their aim is to pull me into something more tangible. So anyways, the dude in ques...

Changing Pace

This blog has been a big part of my life for the last five years. Besides giving me the opportunity to connect with a number of interesting people and share my thoughts and ideas with them, it has been a form of daily meditation for me. No matter what the day threw my way, I made a very deliberate effort to find a little quiet time to write.The process of thinking about what to write and then the act of writing itself worked as an antidote to aggravations big and small. Five and half years ago, when I started Heartcrossings both my personal and professional lives left a lot to be desired for. The only real happiness I had was in being J's mother. While that was often enough to make me forget what I did not have, I sorely needed a third place to call my own and shape in the likeness of my dreams. This blog has been where there were no limits or constraints and that was absolutely exhilarating - it is the reason I have been able to nurture it for as long and as much as I have. A lot ...