It is a profound irony that the more you know about a particular industry, and the more experience you gain in it, the more difficult it can be to move it forward with truly meaningful innovation. But it's true, thanks to something known as "the curse of knowledge" — one of the most vexing cognitive biases identified by psychologists and behavioral economists. (Another big one is "functional fixedness" — a topic I will save for another day). Cognitive biases are very human and arise from our need to make sense of a situation before deciding on a course of action. As we acquire, retain, and process relevant information, we filter it through the context of our own past experience, likes, and dislikes. Not surprisingly, with every subsequent challenge, our response is increasingly shaped by our knowledge of "how we've always done it."
This is part of why open innovation is so powerful. By definition, it sources valuable ideas and inventions from outside the walls of an organization. That not only brings more brainpower to bear on a problem to be solved, it brings minds that are not constrained by industry conventions.
But if you think that by merely opting for open innovation you will escape the curse of knowledge, you may be wrong. Assumptions based on convention can still undermine the effort because, at the outset of any open innovation, someone has to communicate what is being sought.
Made to Stick authors Chip and Dan Heath share a vivid illustration of how the curse of knowledge leads to communication failures. In an experiment, psychologist Elizabeth Newton asked subjects to choose among 120 well-known songs and then tap out the melody with their finger on a table for a listener to try to identify their choice. When she asked the tappers to guess how likely listeners were to recognize the songs, they predicted a 50% success rate. As it turned out, the listeners correctly identified only 2.5% of the melodies they heard tapped. (Newton's 1990 PhD dissertation, "Overconfidence in Communication and Intent: Heard and Unheard Melodies," gives full details.)
Try it yourself — tap away while a familiar tune plays in your head — and you will understand why the answers seemed so obvious to the tappers. In the same way, the knowledge in an engineer's or technologist's head (or a group of them) causes them to make assumptions about what should be clear to anyone, while failing to give outsiders the understanding of a problem that would allow them to solve it in a new way.
Sometimes the curse of knowledge leads experts to communicate what they're looking for at too low a level. Recently, for example, my colleagues and I assisted a large consumer products company attempting to improve its packaging. It sells a perishable product that consumers don't use all at once, so its engineers had identified the need for a better re-sealing solution. But when we articulated the need, we went beyond describing what would constitute an ideal sealing technology; we specified how much freshness and taste quality had to be maintained over what length of time. (Other packaging performance factors such as ease of use and cost were also stipulated.) Being clear that the need was to preserve food quality, not just to seal a package, affected how solution providers approached the problem. The overall set of submissions was of high quality, as judged by how well each met the criteria for an ideal solution. Most important, the search resulted in a new package innovation -creatively combining different approaches to achieve the goal — which was promptly patented by the client.
In another search, the curse of knowledge made an organization communicate its need at too high a level. This was the International AIDS Vaccine Initiative, which decided to sponsor an open challenge to the scientific community to come up with an effective inoculation against that terrible disease. Unfortunately, defining the Request for Proposals as a vaccine challenge did not yield many high-quality responses. Our advice was to break down the need to a level where scientists who did not think of themselves as vaccine creators would engage. More fundamentally, this was a protein stabilization challenge. Once it was refocused on that critical stumbling block, the technology search brought back 34 proposals from highly qualified scientists in 14 countries. Three of these were sufficiently promising that IAVI funded their further development with $875K in research grants. Even for an organization full of smart scientists — indeed, especially for such an organization — it can take a third party, unencumbered by presupposition, to overcome the curse of knowledge.
The vaccine example underscores that if you define a challenge and its ideal solution's qualities and characteristics in an application-agnostic way, you defeat the curse of knowledge in two ways. First, the knowledge of the expert sourcing the solution doesn't translate to limiting assumptions about what form it will take. Potential solution providers are given an understanding of the challenge that doesn't constrain their ideation. Second, the potential solution providers are less likely to self-select themselves and their ideas out of contention because they don't think they're relevant.
I regularly see companies' open innovation efforts being undermined by the curse of knowledge. They write detailed specifications for the technology they are seeking based on what they have seen work in the past. They draw up exclusion lists that automatically remove certain companies or industries, and the science they have mastered, from their consideration. Without even recognizing that they are making assumptions, they contract their universe and discourage viable submissions.
The only way to avoid these missteps is to place a lot of emphasis on how the need for a solution is communicated up front. In open calls for innovation, we need to clearly communicate to others the real problem to be solved and the benefits the solution must deliver, as well as our own understanding of the chief stumbling blocks and the features a solution will offer.
In our role as innovation facilitators, we have to remember that a little knowledge can be a dangerous thing — and a lot of knowledge can be a curse.