Datafication and ideological blindness
“Our bodies break / And the blood just spills and spills / And here we sit debating math.”
—Retribution Gospel Choir, Breaker
Design got its seat at the table, which is good because we can shut up about it now. What used to be seen as the territory of bespectacled Scandinavians is now a matter of HBR covers, consumer clamour, and 12-figure market caps. People in suits now talk about design as a way to differentiate products and unlock new markets.
The table is a metaphor for influence, of course. Designers already have plenty of tactical influence – interface, layout, structure and all that – but this is influence of a different order. It is deep and internal: influence over culture, vision, and most of all strategy, the art of deciding where to go and how to get there.
In this realm, data is king. Whether from device sensors, social media chatter, or experiment analytics, data pours off every surface of the modern world, and people are happy to sell us expensive tools to analyse it.
Data has transformed strategy across many industries. Sports fans and insiders alike have become trainspotters: the minutiae of Moneyball, of take-on percentages and suspension loads are now mundane. Evidence-based medicine has put empiricism at the heart of the profession, with randomised controlled trials guiding new treatments and in some cases reducing mortality.
But outcomes are only half the story. Much of the appeal of this datafication is ideological.
“Quantified thinking is the dominant ideology of contemporary life: not just in scientific and computational domains but in government policy, social relations and individual identity.”
—James Bridle, What’s Wrong with Big Data?
The tech industry believes itself to be neutral and objective. This is pure self-delusion. Ideology runs hot through the veins of the sector. So blown are we by the winds of the New, it takes just weeks for a prevailing zephyr to align all ships in the same direction.
Today’s dominant tech ideology is Lean Startup, a California-ised nephew of Lean Manufacturing. The family resemblance comes in the elimination of wasteful work that fails to meet customer needs. So far, so obvious. In practice Lean Startup almost exclusively manifests as accelerated empiricism.
Lean Startup’s central tenet is that we’re surrounded by unparalleled uncertainty, to the extent that accurate forecasting is impossible. Therefore, adherents claim, the only worthwhile way to build is through stepwise iteration, in a perpetual cycle of Build-Measure-Learn. The notions of intuition and prediction are negated, deprecated by data.
I’m not convinced by the presumption. Certainly the tech industry operates amid flux, but the wide-angle view of this change is more predictable than many would admit. Bill Buxton famously claimed consumer tech has a 30-year ramp-up, pointing to the mouse and the touchscreen, first prototyped in R&D circles in the mid-1960s. Even the Gartner Hype Cycle, tacky as it is, offers a plausible model of trajectory and velocity for emerging technology. With intelligent extrapolation and study, the next five years of technology is hardly a mystery. The second-order and social impacts are murkier, true, but here a spot of science fiction scrutiny and primary research surely isn’t beyond us.
But the message is out of vogue, and a posteriori empiricism is in the ascendancy. So datafication it is, and with a narrow view of data at that. In Lean Startup as now practised, data is first and foremost quantitative, usually gained from user analytics and multivariate experiments.
I’ve studied a good deal of mathematics and statistics, and know the power of quant data. But I also know its limitations, and have seen first-hand the dangers of data ideologies excluding other decision-making inputs.
Scenario 1 – experimentation trumps coherence
I’ve worked with two companies where the primary product strategy has been reducible to “Increase this KPI”. The same sorry tale has panned out in both.
At the start, things look positive. Per executive edict, employees concoct product experiments to move the needle. Pace of execution goes up, pet projects ship, and people are pleased at the rapid throughput and product change. Sometimes the measure does indeed move, and from a distance it certainly looks like innovation.
But almost all these experiments are additive, so the interface gets crammed. White space is eroded by buttons and info. Successful A/B trials ship to 100% regardless of coherence and intent. The product slowly becomes cluttered and the value proposition becomes incoherent. Secondary metrics that lie outside the scope of the experiments, such as retention or NPS, start to plateau, then slip.
Worse, the internal framing of users shifts. Employees start to see their users not as raison d’être but as subjects, as means to hit targets. People become masses, and in the vacuum of values and vision, unethical design is the natural result. Anything that moves the needle is fair game: no one is willing to argue with data.
PMs and engineers decide that since they can ship pretty much whatever they like, they bypass what they see as designers’ obstructive, oversensitive tendencies. Deployment authority becomes the ultimate power, design morale plummets, and designers quit. This proves to be a leading indicator of company morale, and general confidence in leadership sags shortly after. Failure to provide a strategic North Star is itself an absence of leadership; a timid disavowal of responsibility for direction. So the short-term happiness soon fades, and the breakdown of collaboration and strategic coherence proves hard to reverse. Usually you have to sack an exec or two.
Scenario 2 – Safety dominates
In a data-paralysed company, conviction is discouraged. Skills are diminished to perspectives, and only hypotheses have currency: weak opinions, barely held. There’s a shift from fulfilling user need to squashing risk, and heavy conservatism sets in. The symphony orchestra of design is reduced to the barbershop quartet of conversion rate optimisation, and the product hillclimbs to the well-known local maximum. Innovation becomes purely incremental.
Now, there’s nothing wrong with incremental innovation per se, unless it becomes the only way you innovate. In an environment of data-enforced caution, there’s no way to climb down that hill to higher pastures elsewhere: a single metre is downhill, so you’ll never walk a hundred. Companies thus paralysed, unable to take bold steps in new directions, become vulnerable to eventual disruption.
This malaise is particularly dangerous because it’s symptomless until too late. Your outlook seems healthy for many years until one day you’re suddenly irrelevant.
For most companies, deep commitment to product/market fit will prove more valuable than a safety-first optimisation mindset. As Ben McRedmond of Intercom says, a billion-dollar business was never built off better button colours. At vast scale, a 0.1% conversion uplift could indeed mean $millions, but to a company not in that league, premature datafication could be fatal. Better to focus on truly understanding and addressing user needs rather than shaving a tiny advantage in a conversion funnel. Optimisation is the cherry, not the cake.
Scenario 3 – copycat strategies
Replacing strategy with metric optimisation is stupid enough, but it’s even more dangerous for companies that choose the same metric as competitors.
Social networks typically make engagement their primary target, and consider it a proxy for user success. It’s now clear that among the strongest drivers of social network engagement are rich media (images and video), contemporaneity, and easy feedback mechanisms. Little wonder then that all social networks are headed toward the same territory of videos, live streaming, and push-button social grooming. It’s the preordained endgame of a battle for engagement, and so every social network starts to look the same.
A strategy is useless if your stronger competitor has the same strategy. Without differentiation there’s no advantage, so metric-copycat strategies tend to lead to one of two scenarios:
If scale matters (any domain with Metcalfian dynamics, e.g. multiplayer gaming, social networks, two-sided platforms like classifieds or ride sharing), the winner takes all. Any incumbent would love its competition to ride the same rails – it can then leave the risky R&D/innovation to the chasing pack, cherry-pick what works, and roll it out to a wider audience, thus protecting future market share. Why bother checking out the alternatives when we’ve copied the best bits here? Call it a fast-follow strategy if you want to wrap its ethical deficiencies in a cloak of respectability.
If network effects are negligible (e-commerce, publishing, task-based software), cost is the only real differentiator left, and it’s an ugly race to the bottom. Again the bigger player usually wins. They can discount the sharpest and absorb losses the longest, then ramp margins back up once the competition is dead.
Scenario 4 – flawed data, flawed decisions
If you’re putting data at the heart of your decision-making, you need to get it right. That means:
employing skilled staff who will set up experiments accurately, avoid flaws such as p-hacking, and have the numeracy and statistical capability to draw valid insight from your raw data.
investing in watertight analytics technology with excellent uptime and security.
a laser-like focus on team efficiency and deployment. No point garnering insights if you can’t act.
surprisingly large sample sizes. Thanks to the complexities of statistical power, you may find a minor tweak in a low-conversion process will need a >100,000-user sample for a valid test.
Without these, you may be making decisions off faulty data. Worse, you won’t even know. Thanks to the legitimising effect of datafication, you’ll feel highly confident while doing the wrong thing; betting on a hand of four ♠s and a ♣ that you misread as a flush.
Data can of course be an enormously valuable strategic input, if these pitfalls are sidestepped. Senior designers and leaders can’t withdraw from the data discourse, but they are well placed to question its ideological power. Data is a valuable adviser but a tyrannical master, and in some companies datafication has such a stranglehold that other approaches are permanently in shadow.
Fortunately, these companies are easy to spot: they call themselves “data-driven”. Run from data-driven companies. In thrall to semi-science and blinded by their dogma, they’ve lost the ability to see intelligent alternative perspectives on their business, their products, and the world. Embrace instead data-informed companies. This isn’t mere grammatical pedantry – a company genuinely informed by data understands the risks of datafication and adopts sophisticated, balanced approaches to strategy that blend quant, qual, and even some of that unfashionable prediction and intuition.
