Data Analytics with Open Source Tools

A long time data wrangler serving many masters as one must in this role, I have been looking for a book that talked about the real life challenges of the job. I would love some practical advice on how to do my job better without driving myself completely crazy.

I found at least some of that in Philipp K. Janert’s book Data Analytics with Open Source Tools. I am not the right audience for the math in the book and based on my experience translating something that technical to executive management would be extremely challenging if not impossible. Often there are no serious math nerds on the team that understand the concerns of the business well enough to bring their numerical and computations skills to bear on them effectively (i.e. three action items to improve customer engagement by 15% in the next 90 days).

More often than not, it is falls on the rest of us who straddle the technical and business worlds, to divine (or help divine) something of value from the many cesspools of enterprise data. To be successful, we to know how to make the most of what little we have in terms of clean data, repeatable processes, inertia to improving them and a common understanding of data across the enterprise.

In the preface and introduction of his book, Janert advocates using as little statistics as possible, going with the most commonsense way to analyze the data set and get a feel for it just by looking at it. Slice and dice it many ways, run some charts and numbers to see if there is an interesting story buried there somewhere. This is been my approach almost 90% of the time and I was excited to see it endorsed by the author. I have used what the math yielded as a way to prove or disprove my story. While far from perfect, the method has helped point clients in the right direction, remedy issues that would have otherwise gone undiscovered.

Later in the book, the author brings up a very important point. Getting data to be good enough is often feasible but to get it to be truly high quality maybe an impossible task. If the success of a project hinges completely on the data being better than good enough, it may be wiser not to take on the project at all. This is excellent advice that I will remember to pass on to clients who are bent on cleaning the Augean stables in their quest for business intelligence nirvana.

I would definitely refer this book again if my job ever required me to do the math on data instead of analyzing it using the far less rigorous techniques that most shops are content to use. However, I will continue to look for a cookbook for the analyst who has to work within constraints of time, poor data quality and lack of cohesive processes that are the sources of data. Ideally, this book will have case studies, problem scenarios and real-life solutions that folks like myself can relate to and apply on our own jobs.

Saving For J

I was chatting with another mom about saving for kids to go to college. "Have you already started a college fund for J ?" she asked. I did not know how to answer that question honestly. 

I don't have a fund started for her and will likely never have one. Both DB and I agree that it is a bad idea to give children the idea that their parents will bankroll their education. I tell J that she can go to any college that she can get a full tuition ride to.

The final outcome depends on the amount of hard work she is willing to put in. We will support her in every way possible but she is already getting the message that we will not pay her way to a school of her choice.

So, I may never really put aside money for her college tuition. I would also never allow J to take out a big loan to attend some pricey school. DB is willing to let her work part-time to pay for school. I am not sure I can get behind that idea fully. I'd rather she spent that time and energy to secure a tuition waiver - and focus on getting an education while at school. The other mom sounded skeptical about my idea but admitted it was a bold one. Time will tell if I can walk the talk.

Heart's Traverse

January is usually the month, I clean up my Inboxes. This year I have been tardy. This short verse was written in 2004. There are no notes to indicate who and what prompted this. So many years later, I cannot even hazard a guess. Clearly, this is not a road I will be coming down again. There was something bitter-sweet in that realization.

Of all places that you and I will never
be together, the traverse to your heart
will be the most rued. From the summit
of joy to the death valley of gloom there
was a path that led there. You never
told me , I never asked to know. If I ever
come visiting, it will be down that road.

This other one written in 2005 is much easier to decipher. No, this one I have not forgotten - at least not as yet. Yet reading it bring back no memories good or bad. It is like remembering the receptacle and not its contents.

Greetings to you my Medusan muse !
I see your youth reflected in another
who like you is not mine or even meant to be.
He of burnished copper skin, dreamy eyes.
A smile that winnows gloom out of my day
Were you like him once ?
I think  you shear away the wild, unstoppable you
when you trade those luscious locks for a buzz.
You let your words drizzle out like acid pellets
gouging a good earth longing to give.
You said you would move heaven and earth to
be with the one for you. I was she – we both know it.
Yet sea foam did not reach for the planets.
The drooping shell of you  merely walked away.

Continuity Errors

I was driving DB's car most of last week. One evening, on the way back home, the radio station he has it set to, picked up a song I had not heard in a long time. I caught the last few bars, it sparked recollection but not recognition and then it faded out to headline news. I tried to remember when I had heard it last. It was so long ago, that it could have been another life. 

But again, I have a few different lives - the life in India as a young girl enjoying the independence that comes with her first job, a new bride who came to America full of romantic ideas about marriage and high hopes for the future, an emotionally devastated new mother who left that marriage to find her way in the world alone, the determined single mother who raced against the clock non-stop for eight years and now a woman who is trying marriage, husband and home one more time. 

Each phase of my life, I was a person that bore little resemblance to who I was before or after. Music and books have formed the tenuous ties between these lives, selves and phases. Listening to that unidentified yet very familiar piece of music was like watching a continuity error.

Lolita In The Suburbia

I picked up the movie Towelhead from the library recently. Relationship angst and a thirteen year old Lebanese American girl caught my attention. J is going to be a teenager soon and I will the immigrant parent trying to make sense of the life changes she will deal with in a culture that I don't fully understand. I am always trying to learn ahead of time so I can support her better.To that extent, any and all insights into the world that a minority teenager dwells in America are useful to me.

Sadly, I was not able to bring myself to watch this movie past the first thirty or so minutes. The idea of a child (which to me thirteen year old Jasira, the film's protagonist is) could be so vulnerable got me anxious to the point that I could not bear to sit through the rest of this thing. Her mother's boyfriend "helps" her shave her pubic hair and her father's neighbor leaves porn magazines at her doorstep for her enjoyment. The men in her life are either outraged by her blooming sexuality or uncontrollably titillated by it. It is like they have no control over how they react to a thirteen year old's bodily changes. Add to that the racial slights and other forms of harassment the kid has to deal with at school. Depicting the girl in deliberately Lolita-esque light, is the most pathetic excuse to prop the storyline up.Lolita has been done before and there no additional value a brown suburban girl born to an immigrant father can add to that idea.

The artistic merits of the movie are questionable at best. One assumes that Allan Ball is attempting satire in painting this American Nightmare dystopia that is the world of a young first generation immigrant. The movie may have redeemed itself past the first thirty minutes but I had had it with the gratuitous images of masturbation, menstrual blood, father holding up a bloodied tampon before flushing it down the toilet to find out about it.

Myths Of Innovation

Innovation has to be one of the most overused words in the modern cultural lexicon. It is sought ardently where it cannot be found,ascribed where not apropos and most definitely widely misunderstood. The free and loose way we have with this word and all that it stands for piqued my curiosity about Scott Berkun's book The Myths of Innovation.

Berkun does his readers a big favor early in the book, by disabusing them of the idea of the moment of epiphany which causes innovation to supposedly happen.There is no Eureka or falling apple instant that turns someone into an Archimedes or a Newton. Instead, Berkun argues there is a method and discipline to the business of innovation. It is not happenstance, but the result of many years of hard work and persistence with ideas that very few believe in, coming to fruition in unexpected ways.

One of the magic recipes for innovation, Berkun says :

Hard work in a specific direction - The majority of innovations come from dedicated people in a field working hard to solve a well-defined problem. It’s not sexy, and it won’t be in any major motion pictures anytime soon, but  it’s the truth. 

On the kind of mental discipline it takes to to innovate, he says :

The secret to balancing work and play is thinking of the mind as a filter. Instead of binary switches—open vs. closed, creative vs. routine—we need a sliding scale of openness we can control. If you want new ideas, you have to slide toward openness, turning some filters off, exploring thoughts you’d ordinarily reject offhand.

My favorite part of the book is where Berkun outlines a simple plan to turn the promise of innovation to reality. I would recommend The Myths of Innovation to anyone who is curious about the the history and current state of innovation. The many practical ideas on how to innovate in the real world, are a bonus.

Thwarted Education

The events that lead up to my request for a meeting with J's teachers usually have me anxious even before the meeting. At the conference itself, I find it difficult to stick to my rehearsed talking points and before five minutes are up, I am rambling way off topic at furious pace trying to cover all that I have to say in the twenty minutes I have alloted. My frustration levels run so high, that I can't keep it coherent anymore. Each time, I feel like I failed and should have handled it a lot better than I did. This year for the first time, I had DB to accompany me and that was comforting. We had agreed, if I wandered off-course, he would nudge me so I'd come back on track.

I had wanted the time to understand why J's motivation was diminishing. Why it was that she refused to apply herself to do anything beyond "minimally required" to make a decent grade. It took a while for me to even get them to see that there was a problem, that I was interested in whether my child loved learning and found happiness in accomplishing results through hard work. I was not interested in her grades - good, bad or ugly they did not to me indicate how her education was progressing. The moment of truth came when in response to me saying "I don't want my child to become the kid who learns how to make the grades and loses the love and joy of learning. That is what I am afraid is happening to her", the teacher said "This is ultimately not about what you want, but what she wants"

That left me so non-plussed that I could not summon any kind of response. DB was caught off-guard as well. For a teacher with some twenty odd years of experience to say that the course of a nine year old's education and therefore her life is not up to the parent to decide was so stunning that I had nothing left to say. Would she be advocating for J's voting rights next ? Surely, if she can make such significant decisions about her life, she would be well qualified to elect a leader to represent her. 

I send J to school so she benefits from the social interaction with her peers, learns to work in a structured environment and follow instructions. I don't have any academic expectations from the school at all. What I had not counted on was for this enviornment to diminish her innate strengths, make her less than who she is. So not only is the school failing in its primary goal to "educate" because the teachers equate education to grades, it is also harming her by being the enviornment in which her curiosity, creativity and yen for learning are actively thwarted. And this at what is touted to be one of the best schools in the district. 

What was most surreal about the experience was the hubris of the two teachers we met. They were completely convinced that they were doing an outstanding job educating the kids in their charge, that their tools and methods were absolutely the best. If the kid was not thriving, it was up to the kid to figure out how they could do better. Their process was simply perfect and would not be tuned. I realized when people get to that point in their profession when they turn deaf to unfamiliar ideas, there is no hope for them or those whose life their methods impact. So it is with J's teachers and J. In the private sector, dinosaurs such as these teachers are often let go when organizations restructure and get an infusion of fresh blood from elsewhere in the industry or university campuses.

I have come to the sobering realization that along with educating J at home, I am also responsible for making sure that I undo the damage that the school is doing to her before it is too late.

That Right Feeling

There was something about Vir's voice that told Sheila that he may be the one. She had not seen him at the time - they worked in different locations of the same company and had met for the first time on a conference call. She tried to remain focused on the agenda and not get distracted each time he spoke. She chided herself for being irrational and ridiculous but it proved hard - Vir seemed to have some subtle subtext just for her as he went over the list of risks to the project. She was glad when the hour was up and Vir was gone. 

They would meet three days later, he would ask her out to lunch. They would end up spending two hours talking about everything under the sun except business. He would ask to meet her again the next day and the day after. Instead of flying out home on Friday, Vir would extend his stay for the weekend. They would spend those two days together. Sheila would feel that supreme sense of comfort she had never felt with any other man in her life. Vir would tell her that "I knew that there was something special about you when you first spoke in that call. I knew I had to finagle a business trip out this way to see if I was right"

A month later they were engaged. They both agreed this was bordering on insanity- their friends urged them to slow down and spend more time before making this serious a commitment.One Sunday, as she drove him to the airport, Vir asked Sheila "Do you want us to wait some more, continue dating ?" and she said "If my past is any indicator of how that will work, we will probably never get married. I have waited this long to feel that right feeling - there is nothing else to wait for". Vir stayed silent. As he kissed her goodbye he said "Sheila, I am more ready for marriage then I have ever been in my life - I don't want us to wait either. Think about when you want us to get married - I am ready when you are"

Vir and Sheila became man and wife a couple of months later, most of their friends were too shocked to be able to congratulate them from the heart - they felt the sense of disconnect and distance from those who had been closest to them in their single years. Vibha was overjoyed and she was in the minority. Most of them stood by to see what damage control they may need to step in to do once the honeymoon period was over and reality set in.

Reviving Ties

The past year has been about change - often of the cataclysmic kind. DB's coming into my life ending a ten year drought in love was as welcome as it was challenging. It was the year when some friends who had supported me for close to ten years as I flew solo, decided that our friendship had run it's course. So I got used to not hearing familiar voices on the phone, not seeing their emails and in time learning to forget that they were once an important part of my life. When the new marriage hit a bump on the road, I had no one to turn and talk to. Instead DB and I sulked in our corners and came back together when we were done.

I was forced to cleave my life in two - yet again. Life before DB and life after. It was reminiscent of when R (my ex) and I parted ways years ago. Spending New Year's eve with my old friend E was very poignant. She met DB for the first time and J after four years. We felt just as welcome as a new family as I did when it was J and I. DB and E got along wonderfully. We talked, ate, drank and took long walks along the nature trail that runs behind her house and runs all the way to the ocean.

Until reconnecting with E, I had been coping with having to start over one too many times in my life. Mourning the loss of friends - DB's lost plenty on his end as well after our marriage. As a couple, we often floundered on our own without the benefit of nurturing friendships. I am meeting my friend V for lunch today. She texted me this morning wanting to meet for lunch - just her and I. We have in the past meet in a larger group, V, K and his family and the three of us. I gather V needs some time alone with me. This is not a revived tie but a tie that needs some strengthening. I feel good about the year ahead.