Whether you are a manufacturer or a retailer, a mid-sized company, or someone with a small business that caters to various customers and clients in a niche market, understanding consumer behaviour is vital for your business to succeed.  Traditional market research or Big Research is the most often used tool for that.

With research, marketers aim to extract helpful data from consumers for predicting behaviour and future preferences, and to plan their brand accordingly.

Research is a useful tool, yes, but it often becomes a crutch. Using research data to inform every aspect of a product, brand, or campaign may seem like a scientifically-sound idea, but it is very dangerous in practice.

Market research, and the information it uncovers, is inherently flawed. It’s flawed because what research asks respondents to do – to rationalize the irrational – is nearly impossible. The factors that affect what we do and why we do are mostly unknown to us and impossible to put into words or rate on a 10-point scale. Research asks us to post-rationalize our actions, assigning sensible reasons to our past decisions, whether they actually played a part in them or not.

And of course, there is the classic question of “How many responses do you really need?”  This simple question is a never-ending dilemma. A larger sample can yield more accurate results — but excessive responses can be an expensive affair. On the other hand, a very small sample size may not represent the universe.

In the mid-1980s, the Coca-Cola Company made a decision to introduce a new beverage product. The company had evidence that taste was the single most important cause of Coke’s decline in the market share in the late 1970s and early 1980s. A new product dubbed “New Coke” was developed that was sweeter than the original-formula Coke.

Almost 200,000 blind product taste tests were conducted in the United States, and more than one-half of the participants favoured New Coke over both the original formula and Pepsi. The new product was introduced and the original formula was withdrawn from the market. This turned out to be a big mistake! Eventually, the company reintroduced the original formula as Coke Classic and tried to market the two products simultaneously.

Ultimately, New Coke was withdrawn from the market.

What went wrong?

There was a flaw in the market research taste tests that were conducted: They assumed that taste was the deciding factor in consumer purchase behavior. Consumers were not told that only one product would be marketed. Thus, they were not asked whether they would give up the original formula for New Coke.

In recent times, the growth of information technology has transformed market research, with a growing number of analysts learning about consumer preferences and buying habits by mining massive sets of quantitative data and employing complex algorithms to uncover patterns and correlations that enable more effective marketing. Data mining often gives businesses enormous amounts of information about their customers’ behaviors and buying habits, enabling them to do effective marketing.

For example, through data mining, online retailers these days have features that tells a potential customer that people who like one particular product also like certain other items or recommend products to them based on their behaviour on the site.