PD fallacy #6: experimentation has no role in product development

Some things are simply unknowable until we try them out. The response of customers to new products or new features is one of these.

In most of the companies I work with, the decision to develop a new product is based on an assessment of the likely revenue and margin the product will generate. This assumes a clear specification of the intended functionality as a basis for the cost estimation associated with the development. Typically, this means product management working hard on creating a case for the product including the differentiating functionality, the position in the market, the intended customer, and so on.

Although this all makes sense in theory, in practice this approach is simplistic and fails to deliver on expectations in most situations I’m aware of. One of the main factors I’ve written about in several earlier posts is the inability to make accurate long-term predictions. Many claim that their organization can, but in my experience, this is mostly about padding the initial estimate to get it to sit at the end of the bell curve so that the likelihood of staying within the estimate is very high. Informally, this is referred to as the rule of pi: you ask for an honest estimate, multiply it with pi (3.14) and use that for planning.

Despite all the best efforts, many product development efforts run over budget, both in terms of time and cost, and that’s where R&D bashing easily starts. Armed with the perfect knowledge of hindsight, senior leaders outside of R&D openly wonder why the idiots in R&D are again late and why we keep allowing those monkeys to destroy the company’s profitability.

Of course, R&D gets its chance at revenge when the product is finally done and in the market, and sales is unable to generate the revenue that was presented at the beginning of the development initiative. This then leads to sales complaining about lacking features and poor product development practices resulting in a subpar product. This is where the finger-pointing starts and people dig their trenches and start lobbing grenades.

To me, the root cause of all this is a fundamental misconception in the heads of most leaders, ie that it’s actually possible to predict what a product should contain in terms of functionality to be successful. This brings me to the distinction between what’s knowable and what’s unknowable.

As the name implies, knowable knowledge is that which can be uncovered simply by putting more energy into collecting data and information. When product development fails due to a lack of insight into knowable aspects, we have a problem in that someone might not have done as good a job as the person should have.

When it comes to unknowable things, the only answer that we have is experimentation. We have to try it out in the hope that we learn what we need. The idea that some things can’t be known before the start of product development and require experimentation during development or after shipping through DevOps-style practices is alien to most companies I’m aware of.

Of course, some aspects might be knowable if we would invest prohibitively large amounts of resources. In those cases, the use of experimentation may be a much better way to collect the necessary information than to sit and guess.

The challenge is that when people are asked to provide tangible answers to unknowable questions concerning the functionality of a product, the most reasonable approach is to simply guess. The problem with guessing is that something that the individual initially understands is just a guess rapidly becomes a truth that’s treated as being cast in stone. People love certainty and hate uncertainty and hence even the most “putting a stake in the ground” guesses quickly become requirements.

One reason for the reticence toward experimentation is that it introduces uncertainty and risk. If we’re unable to determine the exact functionality needed for a new product, we can’t do the effort estimation. If we can’t accurately predict the required effort, we don’t know the expected revenue and margin. All this makes us poor leaders as we decide on product development efforts without proper financial justification.

Still, for all the difficulty of dealing with uncertainty and risk, it doesn’t change anything about the reality of product development. In my view, many of the dysfunctions in product development organizations originate in the inability of leaders to accurately distinguish between knowable and unknowable things. It takes courageous leadership to break out of this conundrum and confront reality as it is, rather than as you wish it to be. Even if it’s much more comfortable to pretend that you know what you don’t, nothing good ever came from ignoring reality.

The best way to address this situation is to treat product development as an iterative process of risk reduction. Risk can be technological in nature, but companies already know how to deal with technology risk. The concept of technology readiness levels (TRLs) was developed as one model to deal with this. The market or customer risk, however, is typically far more significant, but we’re much less well-equipped to deal with this.

Iterative development breaks up product development efforts into a series of decision points where the team is asked to clarify and deal with one or a small set of open questions concerning the product. Each iteration receives a small amount of funds, performs tasks to answer the highest-priority questions and based on the data that comes back, the governance team decides whether to fund the next iteration or to stop development.

One concern that’s often raised when I suggest this approach is that the effort required to even create a simple prototype for testing with customers is so prohibitively expensive that there’s no alternative but the traditional product development model. Although I appreciate that this may be the case in a limited number of cases, in the majority of these, my experience is that it’s more a lack of creativity. We saw the same in software companies adopting Agile practices where initially teams complained about not being able to break down features such that these fit in a single sprint.

The use of minimal viable products and A/B experimentation is typically non-existent and discouraged in traditional companies. The specification is used as a basis for all development activity and there’s no desire to question it for a host of different reasons. This brings us to the notion of knowable versus unknowable. Some things are simply unknowable until we try them out. The response of customers to new products or new features is one of these. This requires a different, more iterative, product development process where each iteration is decided upon once the data from the previous one justifies continuation. This fits hand in glove with digitalization as product development continues after the product has reached the market with the continuous deployment of new functionality. So, in that sense, the iterative approach is a constant throughout the entire product lifecycle and the start of production is just a small blip in the overall process. As Mark Twain said, continuous improvement is better than delayed perfection.