Saturday, March 19, 2011

An Entrenched Offshore Engagement

I am back consulting for client after a hiatus of three years. The last time around, they had just made a major push to outsource their software development work. The old hands in the company were still around, took pride in the products they had built and were accepting the new order grudgingly at best. The offshore team was expected to deliver high quality and when they failed, they were excoriated to put it mildly. If the B team did not cut the mustard, the vendor made sure they replaced them with A+ players.
Someone like myself was able to work with the customer and help deliver a durable solution that met their business needs. It was a long and frequently aggravating process to reach the finish line but we still made it - the technical lead was typically an employee who functioned as the architect and oversaw the delivery. I remember sitting through some very brutal code review sessions after which forty to sixty hours of work was trashed and redone.That was then and this is now. 
The transformation that has taken place in the last three years is unbelievable. Many of the former tech leads have been let go. The few that remain have assumed more managerial positions. Some of them overseeing upwards of forty development resources. They are stretched too thin to provide any useful technical direction. When they are able they have to deal with resources who simply do not have the technical chops to execute on their direction. Used to be that the offshore resources were held to a high bar - made to redo work until its quality was up to par. Not so anymore. They operate in drone mode with no technical leadership from either the client or their own organization. To cut costs, business analysts have been dispensed with. Developers with two to three years of industry experience work directly with business customers to deliver a patch work of bad solutions that is always ready to fall apart. Defects have become hydra headed monsters. Fixing one, opens up ten others and the cycle never ends. Fire fighting is the new business as usual.
The vendor bills the client for the hours and the hours are astronomical. I recently wrapped up another consulting engagement where the client had their entire development crew in-house. Mostly full time employees and a few local senior consultants with niche skills. The IT shop cost this client ten times more than what it does my current client but when you compute the first time right stats along with earned value metrics with regards to schedule and cost, they come out far ahead even with their high priced resources. The estimated hours for any task were within ten percent of the actual hours. My current client routinely sees overages of two hundred percent. Schedules are mostly ornamental with projects habitually going over schedule by six to eight months.
That is the cost side of the story. The far more tragic one is that of quality of the product delivered and the business customer's willingness to settle for any piece of trash that is delivered to them in the name of a system or an application. User experience is a non-existent concept as are stability and performance. Some of the applications have interfaces that remind me of primitive client server applications from the early 90s yet the tool set being used is absolute state of the art. It never ceases to amaze me that they are able to go that far backward in aesthetics and functionality. Each time there is a crisis (which is several times a week), it is all hands on deck to band aid and stop the bleeding. When enough damage happens they decide to lop off the limb and hope that the rest of body can remain functional. 
I was brought in to shore up a highly dysfunctional team and while the I have had success in covering some of the most gaping holes, dysfunctionality is far from gone. I am beginning to realize the problem is likely not a solvable one. The customer no longer knows that they deserve better, that  the quality of what they being served is third rate (if that) or that analysis must always precede solution development. They have come to accept and indeed support the modus operandi of band aid as long as possible, hack limb and sew in a new one when it becomes available. Repeat cycle ad nauseum. It used to that the customer was discerning enough to know that they were not being served well. They realized there was value in pausing for analysis (even if it took a couple of days) to be completed before a defect was fixed. With all that gone, the vendor has no incentive to change the way they do business. Those of us in the metrics business, are expected to provide analysis on root cause of issues, identify process gaps but any recommendation on remediation is usually met with apathy and inaction. It is  enough to have a system that would identify the guilty party and assign blame.Since it is the offshore vendor's show all the way, no matter how you slice the data, they are always at fault. With that, life goes on after statuses have been reported up, executive summaries presented and the de rigeur hand wringing and finger pointing done. 

It has been quite an eye popping experience for me to see what an entrenched offshore engagement looks like and should serve as a cautionary tale for those who  have recently embarked along this path.


Anonymous said...

I think the quality of work delivered by the outsourcing company is dependent on the the quality demanded by the clients.
If you look at big financial institutions, all of them outsource the work to off-shoring firms. But they pay top dollars and demand high quality work. Most of them are happy with the work and have increased offshoring. One reason for that is IT infrastructure plays a critical role in Financial services.
Also, for some business IT systems are not that critical. So they can live with lower quality IT apps or infrastructure. The reason is simple, it does not drives revenue. The idea that all IT systems should be well designed has a little technical bias to it. For most managers, if poor quality does not impact revenues, they would be more than happy to live with poor quality and pay less. It is all about making strategic investments and for lot of businesses IT is not one of them.

Heartcrossings said...

Anon - This is a financial institution I am talking about and the tragedy is they (the client) no longer have the ability to distinguish quality from trash. And yes, there is a revenue impact - and hence the all the hand wringing over metrics.