More than 50% of Big Data Analytics Projects Fail.
In the article, ” 3 Questions to Invoke Magic“, I wrote about the Gartner report which stated more than 50% of Big Data Analytics fail, and the three questions that should be asked, which might prevent some of those failures.
In this article, I want to talk about a key reason that can cause these failures – over-investment, and under delivery. In other words – creating a project that’s “Too Big Not to Fail“.
How can we avoid this?
Let’s answer that by first asking how we got there in the first place. Let me first ask a question, which seemingly is of little relevance to the topic.
Did Kodak see it coming?
Kodak’s journey to oblivion is a popular story. It usually gets portrayed as a case where disruptive technology killed a sleeping incumbent. Most articles that cover the story have a similar tone – Kodak did not see it coming, and stubbornly insisted that digital would never be equal to the quality of film cameras.
The surprising fact is that Kodak did see it coming. In fact, they took action on it as well. It is another matter that the actions were misdirected. Does anyone see Kodak as an innovator in digital technologies much earlier than most competitors? I certainly did not. But they were! Consider the following digital technology firsts by Kodak:
- The first ever CCD (0.1 Megapixel) image sensor, 1975
- First Mega-pixel sensor, 1986
- The first digital SLR (a modified Nikon F3), 1991
But it does not end there. In 1996, Kodak CEO George Fisher clearly saw the writing on the wall. He convinced the board and aggressively pushed his troops – and invested more than $2 Billion in R&D for digital imaging.
The problem? Digital cameras were not mainstream until 2000 – the market was still evolving. They spent all those Billions on an uncertain market – and they took decisions which locked them in. They committed to products which were difficult to change! For example – Kodak installed 10,000 digital kiosk in partner stores. In other words – they bet the farm, and they lost it.
When Trying Harder is a Problem
In their May 2002 Harvard Business Review article titled “Disruptive Change: When Trying Harder Is Part of the Problem“, Clark Gilbert and Joseph L. Bower mentioned that an organization need to see changes as a threat and an opportunity. Here is a gist of what I learned from it, and it is relevant for any firm considering new disruptive investments.
When a disruptive new trend is identified by an organization, they react in multiple ways.
- They could make aggressive investments (like Kodak), and possibly get locked into an evolving and unproven business models.
- They could ignore it, and risk missing the opportunity altogether.
- Or they could make staged investments, and watch as markets evolve.
Now, let’s look at someone who got it right.
Neither Rain nor Snow
Anyone remembers mails? Yes, the one we derogatorily call “snail mails“. Apparently people still send them. In fact, this is the high season for postal deliveries (Thanksgiving and Christmas). USPS estimates delivery of 475 Million packages this holiday season.
As of today, the complete process of reading the envelopes, routing them to the correct post office and filing it to the appropriate carriers’ stack is done by a computer. Obviously, it wasn’t always like that. The most complicated part f the process is recognizing handwritten addresses – an analytic job. After 4 years of research, the USPC implemented its first handwriting recognition program in 1987, during the Christmas season. The initial prototype when deployed, could not scan 85% of the envelops. And of the 15% it scanned, it could correctly identify addresses for only 10%.
Sounds like a remarkable failure!
That project should have been axed. Luckily, USPS saw this in advance. The project was developed along with University of Buffalos’ Center for Excellence in Document Analysis and Recognition (CEDAR), with an optimal investment strategy. Even with 10% success, it still saved USPS several thousand dollars that Christmas alone (as claimed). It was indeed seen as a great success. From that humble beginning, things progressed pretty fast. As of 2012, the claimed success rate of their address scanning is 90%. (Do keep in mind that USPS loses anywhere from $700 Million to $2 Billion per quarter these days. But that is the story of a different disruption).
This is a tangential, nonetheless it is well worth taking a minute to marvel at the remarkable pattern recognition machine called human brain. Think about it – even now, with the incredible computing power we have – not many machines can beat a human at handwriting recognition. How about recognizing that cute little cat in a Youtube video? Google’s ‘deep learning’ (a fancy name for many of the semi-supervised machine learning techniques) algorithm and and array of ‘just’ 16,000 computers can find it with a 78.4% accuracy. That is easily bested by a 3 year old kid. All thanks to a part in our brain which weighs less than 10 grams – the anterior hippocampus.
Start Small, Win Early, Win Together
What does all this mean for Big Data Analytics investments? Here is my takeaway. Unless your core business is data analytics – don’t bet the the farm on it. Do not go and ask for that $20 M budget, and do not create a vision that will start producing something 2 years from now.
Here are the seven recommendations. I would put them under two logical groups.
The first four is all about setting realistic expectations, and aligning with the goals of the organization.
1.Start Small:Start with a few clear goals, but be sure to have a grand vision. Success will come soon enough. JK Rowling mentioned she envisioned all seven books at the start. Your magnum opus should be no less.
2. Show Early Wins: Two years from now is too late. A lot changes between now and then. Implement quick wins, and make sure everyone sees the value!
3. Have a Laser Sharp Focus: It is easy to get distracted and be everything to everyone. Soon, they would want you to do enterprise BI and miscellaneous reports. That impacts you in two ways – it diverts your attention, and it antagonizes the business units who are churning out those reports.
4. Align to Company Goals: The best results will come if your analytic goals have a clear alignment to your organizations’ goals and core competencies. If its Sales, try to use analytics to improve sales further. If it is Logistics – use telematics/ Location Based Services / xyz to improve that further.
Peter Drucker famously said “culture eats strategy for breakfast”. The best strategies will be washouts if they don’t resonate with the company culture. Thus, our next three recommendations will be focused on the getting the cultural aspects right.
5. Develop a Culture of Sharing:In this case, we mean data. Business should be encouraged to share data freely among units. In fact we have invented corporate data philanthropy as well to enable organizations to share data for a better society. Caveats of privacy, anonymity and security are applicable.
6. Make Everyone Win: If you see this as a win only for a centralized analytics organization, it is a sure-shot recipe for failure. A lot of insights should come from lines of business, and they are responsible for implementing the analytic insights. Make sure they are motivated to do it.
7. Have Skin in the Game: If it is run as pure IT project, the investment and risks are not evenly shared. The increasing IT budget will put you in the backseat, and you will be justifying costs to everyone. Make sure Line of Business’ have invested their budget.
A final tip: If you are an engineer evangelizing big data/analytic solutions never underestimate the power of marketing and sales skills. I learned the hard way that those two skills are a must for any person in any profession. Acquire them, and your career will thank you.
In Closing…
That sums it up. I thank you for reading my post. I sincerely hope you found it beneficial.
