One thing interesting #12: Arguing against numbers

Numbers are exact. But when they're used to represent the use of a product, they become an abstraction of the actual use.

Sure, if an analytics report says 78% of people tapped on a button; then we can pretty much say, with near-absolute certainty, that 78% of people tapped on that button.

But that's about it.

What about all the other significant activity surrounding this monumental button-tapping moment? What else happened in the milliseconds before and milliseconds after? What about the things we didn't record? What about the things we can't record? What's going on in this persons head as they're tapping this button?

For all the power of the analytics systems in our digital and web products, we can never completely - and quantitatively - model the decisions humans make in the real world.

Numbers can fool even the smartest of us in to think we've obtained complete understanding because of the binary outcomes numbers can determine for us.

To compensate, we need qualitative data.

Qual data is a magical short cut that avoids the infinitely complex task of modelling the real world.

It also makes me not sad.

Qual data is only useful when human brains do something with it. Which implies that our own judgement and intuition aren't things we can delegate to machines just yet. Putting the advent of generalised AI aside for now, it's reassuring to know we're allowed to have confidence in our own abilities and use skill to decide what to do next. (Be 'creative' perhaps?)

Many find comfort in numbers but it's dangerous to rely on them alone. Similarly, those allergic to spreadsheets can often become too dogmatic about the opposite. An approach which runs the risk of making decisions based purely on 'anecdata' or assumptions.

What I love about research and understanding users is that - to do it well - requires a perfect blend of two very different skill sets. Two different modes of thinking - analytical and intuitive - are required to really uncover the truth about something.

I really like this story Jared Spool and Chris Callaghan tell about a train ticket ordering website in this short-but-sweet podcast. By simply 'speaking to some humans' and getting them to use the product much greater understanding was obtained. All the super-sophisticated use of fancy analytics tools might have felt like 'proper work', but, in the end, the more efficient way tackle the problem actually turned out to be very unsophisticated

Spirits, Claws, and Analytics: A study in superstition and science - UIE.FM