Data driven product management is a fallacy

There have been many product leaders who advocate data-driven product management - but personally, I think it's a fallacy.


Always leading with the data (the quantitative, computational driven, cold hard numbers) creates focus on the what, and misses a crucial perspective - the why.


Don't get me wrong - quantitative metrics are a really really useful tool within the product managers tool kit. But being solely focused on increasing conversion per se will only get you so far.


Here's a practical example - let's say we run an A/B test for an online car marketplace, testing a new purchasing journey. The new variant has increased conversion by 5% and increased customer satisfaction by 10%. Sounds great right! In a data-driven world, we would roll this new version out.


But why are more people converting?


Did we put 3 buy buttons on the product page? Does the new design try to recreate scarcity ("7 people have bought this in the past hour")? Using dark or shady UX patterns to encourage users to perform a certain action is a big no-no for me.


There might also be an example where conversion drops after an experiment, but rolling out the losing variant might be the best option. The failed experiment may provide greater benefits further down the line (it could be a pre-requisite to a bigger piece of work, or include new branding/UI which users could take time to adjust to). The team over at AirBnB discussed something similar on a recent blog of theirs, focusing on the winners curese and why the results of tests don't always add up.


Personally, the best way i've found of getting around this is just speaking to your users (being empathetic and inquisitive). It sounds obvious, but applying some structure to how you speak with your users really makes a huge difference. A few example considerations are below:


  • What is the point of the research?

  • Why are we doing it now?

  • What form of research is most suitable?

  • What is the measure of success?

  • Are there any assumptions or hypotheses we want to test?

  • How we can correct against any biases we or our users may have?


When talking to users, listen to not just what they say, but how they say it. Pro tip - this is difficult in a post COVID world, but try to observe your users where they actually would use your product, in the same environment. Zoom calls will get you so far, but seeing it being used in situ will definitely highlight more opportunities.