While Wall Street is considered by some to be the epicentre of ‘big data’, the digital media space has spawned innovations in data processing and analysis that are far more significant.
This is the opinion of Mike Driscoll, co-founder and CEO of Metamarkets, a company providing programmatic analytics solutions for names including Twitter, Millennial Media, LinkedIn and Sharethrough.
And while the New York Stock Exchange comparison seems an impressive one, in actuality programmatic dwarfs the stock markets in terms of the volume of data being transferred.
In light of this perspective, PerformanceIN caught up with Driscoll to tap into his knowledge of programmatic trading, its unrealised scale and where the industry could be heading in the foreseeable future.
The amount of activity in the programmatic space is reminiscent of the activity on the stock market, but only after years of maturation. Why has come about so quickly?
Mike Driscoll: It’s a combination of two major factors in the last 5-10 years: the widespread availability of powerful, ‘big data’ processing technologies such as Hadoop, and the explosion of affordable smartphones. Hadoop made it possible to deliver data-driven ad campaigns at scale, and the phone provided a channel to reach billions of consumers wherever and whenever.
What innovations in data processing analysis were needed by marketers to spur this transformation?
MD: Applications used to run on single desktops or servers, but today nearly every web service we use, from Facebook to Gmail, is spread across a vast array of machines in the cloud. The programmatic advertising markets likewise involve armies of servers managed by thousands of distinct companies. Making these systems work has required innovations in how to distribute data and parallelise computation, which open-source frameworks such as Hadoop and our Druid database address in different ways. In terms of bandwidth and processing power, the distributed systems behind programmatic represent the largest supercomputer on earth.
How quickly do you expect the next technological innovation to be realised?
MD: The next technical innovation is already here: streaming data processing. Hadoop is powerful, but it’s batch-based, which means you have to load data into it in chunks and wait for it to do the work, which as the data gets more complex, can take hours or days. Streaming technologies such as Spark, Kafka and our Druid process data as it flows in, billions of events per second. This allows you to ask complex questions and see the results immediately. It’s a game changer for people managing live campaigns and exchanges, and for marketers who want to understand how all their partners are performing in real-time.
How long is it before the activity in this space eclipses financial markets by a wide margin?
MD: It’s already happened. There may be far more money coursing through the veins of the financial system, but the volume and complexity of data flowing in programmatic is orders of magnitude bigger. By our estimate, there are about 400 billion ads bought programmatically every day - compare that to 7 billion - which is the record for most number of shares traded in one day on the largest stock exchange in the world: NYSE.
How important is it for marketers to better understand the activity taking place in the programmatic space?
MD: The programmatic ad space is the marketplace for human attention. There’s never been anything like it, and all digital advertising is moving in this direction. If you’re a marketer that’s serious about out-maneuvering the competition to reach consumers, the transactional data in this space is a critical piece of the puzzle. You really have to approach this space with the mindset of a hedge-fund manager, and the savviest brands know this.
Will this continue and what does this mean for the industry?
MD: Programmatic markets have given the world a massively scalable, low-cost, trackable transactional fabric for digital content and goods. The promise for marketers is a dramatically lowered cost of campaign delivery and, for the first time, true transparency into what they purchased. We’re seeing growing pains, but marketers are demanding - and they will soon receive - an itemised receipt for every impression they buy.