Big vs Small Data: Is Bigger Always Better?

If you were to rank the significance of a concept by its popularity and ability to spread across sectors, ‘big data’ must surely be among the most important of recent years. The explosive interest in the concept is reflected by Google Trends, which shows a steep hike in searches, starting around 2010-2011:

Data or insight?

But as marketers, whether at brands or agencies, we’re of course looking for ways to turn this data into insight. And it’s here that big data’s ‘bigness’ can be a weakness as well as a strength. Gartner’s famous ‘3 Vs’ definition of big data – ‘high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery, and process optimisation’ – neatly encapsulates the problem.

These ‘new forms of processing’ can take time to perfect, and are often more taxing, in terms of time, labour and technology than traditional methods. Just as in the early days of CRM, businesses had to work out how to track the customer journey across departments, and using disparate information sources, so big data sprawls, by definition, outside of existing analytical and reporting channels, making it harder to analyse.

Last week’s announcement that IBM Research Labs unveiled a prototype device they claimed could break the speed record for information transfer reveals a central paradox of big data, at least as applied to marketing insight. While improvements in technology mean that storage and retrieval speeds are likely to increase exponentially, the rigour required to deal with this variety of disparate data sets will require a longer period of analysis, using as-yet unperfected methods. Essentially, while the data is fast, the insights can be slow, when compared with the agility of insight and decision-making when you have a team of specialists dealing with a single data source e.g. a paid search campaign report.

Small data can still generate insight

At times we as marketeers can be guilty of becoming obsessed with the latest ‘silver bullet’. Big data has fantastic potential for insight, but that doesn’t mean it is required in every situation. I would argue that imaginative analysis of the ‘small’ data we can immediately access will often have the greatest payoff.

By ‘small data’ I mean that drawn from our regular analytics and CRM. This may be insufficient for more involved work but is often sufficient for everyday decision-making, and it has the advantage of being readily accessible and digestible. In many cases, we can get more out of our existing analytics. Too often, analytics is treated solely as a reporting tool rather than an insight platform.

Reminding ourselves that ‘small data’ remains a proven ground of valuable insight will help us in the long run – not only to make better use of our  resources, but also to identify those issues we really do need big data to help us solve.

Again, it comes back to the distinction between data and insight. A number of commentators have flagged the issue of ‘data confusion’, or what we might equally call ‘data fatigue’. It’s the classic wood-for-the-trees situation, where the sheer abundance of data actually makes the process of sifting it for actionable insight much more difficult.

Not opposed but complementary

Finally, it’s worth remembering that big and small data are not opposed but complementary. In fact, big data depends on small data. After all, big data insight is an agglomeration of correctly interpreted bits of smaller data.

Even across digital channels – supposedly the most susceptible to purely quantitative analysis, we still need experts to tease out nuance, sense check, and give meanings to the numbers; whether that’s the ebb and flow of sentiment in a Twitter discussion, or the customer intent behind a particular search query.

Speaking of big data, then, as some sort of replacement for traditional analysis is both to miss an opportunity for insight, and to misunderstand what ‘big data’ is and where it comes from. Ultimately, it’s a case of the right tool for the right job.  The point when all marketeers can make informed judgments about the depth and breadth of data required to answer a particular business challenge, and know where to find it, will mark the maturity of big data as a marketing tool.