Archive

Archive for March, 2009

Listen to your Customers, not the HiPPO (Highest Paid Person’s Opinion)

March 24th, 2009

A while ago, Christopher Nguyen pointed me to an excellent online article that I skimmed quickly at the time and thought seemed sensible.  Its appeal to me has increased over time and I just re-read it with the care that it deserves.

Link to the paper, “Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO,” and a nice overview presentation.

Dont Listen to the Highest Paid Persons Opinon, Listen to your Customers!

Don't Listen to the Highest Paid Person's Opinon, Listen to your Customers!

Of course, the paper and presentation are about the specifics of how to implement good A/B testing in an online world, which is clearly applicable to mobile sites/apps as well.  But, even more importantly, implicit in this way of thinking is a complete change from the old engineering approach of yesterday.

We used to start with an effort to collect inputs to create a Marketing Requirements Document, then a supporting Product Requirements Document would be created, then an Architectural Document would be created, then GANNT charts and job assignments were issued, then work would actually begin, and eventually a work product would come out the back end of the process.  We would then pray that the customers would actually buy it in the volumes we had hoped–usually they didn’t, so we would attempt to collect feedback from prospective customers in a very subjective and error prone manner.  We’d then do an iteration on the process above, culminating in more praying that the customers would buy…

The world is different today.  A properly constructed business assumes that first few iterations will fail, so it

  • seeks to minimize the time and cost associated with getting something to market and with each of those cycles of learning;
  • knows that every customer click is a survey response, and the resultant data trumps intuition about what the customers might do; and
  • tries many diverse ideas, because it is impossible to predict a priori what will work well.

This is good news for getting new offerings out into the world quickly and cheaply.  It is clearly initially more applicable to consumer offerings than enterprise offerings, but that will change over time, I believe.

Uncategorized , , ,