Ten Bullets for Industrial Research
Several years ago I stumbled onto Tom Sachs’ Ten Bullets - his principles for a happy and productive work environment. Sachs inspired me to come up with my own list for a high-performance research organization. The following list has evolved over time as I encountered activities or behaviors that hurt productivity or damaged morale. The list is not comprehensive, but I believe these are ten high-impact principles for effective research.
Disagreement is not disrespect
Disagreements will occur as a natural consequence of the research process. In unhealthy organizations, the resultant conflict leads to mutual distrust and suspicion and causes significant knowledge waste and unnecessary friction. In healthy organizations, conflict drives information exchange and helps build consensus. Keep in mind that it is difficult to learn anything new by only listening to those who agree with you. Speak with data so that disagreements do not become personal. Better yet, stop debating, get out of the office or conference room, and run the experiment together. The feeling of being in a great culture is not smoothness or lack of conflict – it’s the feeling of solving hard problems with people you admire.
Be a “learner,” not a “knower”
Innovation comes at the intersection of ideas, and good ideas can come from anywhere. A “learner” is someone humble enough to abandon ownership of their ideas, seek input from those they don’t always agree with, and then run the experiment. Carol Dweck calls this a growth mindset. Learners tend to embrace challenges, persevere in the face of adversity, accept and learn from failure, focus on process rather than outcome, and see abilities as skills which are developed through effort. They are more likely to see effort as something that propels learning and to see feedback and failure as opportunities to build new skills. A “knower,” on the other hand, is heavily invested in their own ideas, rarely seeks input from others, and believes their status and credentials ensure their authority. If you believe that your qualities are unchangeable — what Dweck calls a fixed mindset — you will want to prove yourself correct over and over rather than learning from your mistakes. To be a good problem solver, you must adopt the mindset of a learner. Moreover, skilled learners avoid the temptation to solve other people’s problems for them. Helping people solve challenging problems on their own represents one of the highest forms of respect.
Beware of confirmation bias
Confirmation bias is the tendency to pay attention to data that supports your view and ignore everything else. Confirmation bias is a powerful human tendency, and we must always be on guard against its influence. We have to actively recognize our biases and temporarily suspend them in our mind as we take in new information and external data so we can be as objective as possible. It is a fallacy to think we don’t need to do this actively, that we’ll naturally do it passively. Recognize that you don't know everything and that your beliefs may be wrong. Be open to learning new things and changing your mind. You stand a better chance of overcoming bias if you tackle an innovative project as part of a team rather than trying to do it alone. Working as part of a team exposes you to people who have different beliefs than you and can help you to see the issue from a different perspective.
State your hypothesis
Richard Feynman describes the scientific method as
Observation – the gathering and recording of data about the natural world
Reason – the ability to think, understand, and draw conclusions
Experiment – a test carried out to see how something works
For an experiment to be scientific, it must be possible that the hypothesis will be refuted. It is extremely important that you write down what you expect to happen before you conduct an experiment. If there’s no written prediction, then your brain can trick you into thinking that the results are what you expected (confirmation bias). Learning most often occurs when there is a difference between what we expected to happen and what actually happened.
It is important to remember that uncertainty is normal in research. We cannot predict the future, nor can we think our way to the future. We can find the path to the future only by repeated application of the scientific method. We must move beyond testing to distinguish good from bad. We must structure our experiments to distinguish understood from not understood. Also, be aware that an unexpected observation made during an experiment cannot be used to validate itself. The new observation must now be tested on its own.
Avoid jumping to solutions
When solving a problem, our natural human tendency is to “jump to solutions.” We assume we know what the problem is without seeing what is actually happening, we assume we know how to fix a problem without finding out what is causing it, and we assume we know what is causing the problem without confirming it. To be better at problem solving, we need to help ourselves see what is actually happening and what we actually know. A powerful tool to prevent jumping to a solution is to follow a problem-solving method. Following a good method forces you to truly understand the problem before taking any action. John Shook devised the following method based on his observations of Toyota leaders:
Review the background – what are you talking about and why?
Grasp the current condition and state the problem – where do things stand now?
State the goal or target – what specific outcome is required?
Analyze the root cause – why does the problem exist?
Present countermeasures – what do you propose and why?
State an implementation plan – how will you implement the countermeasures?
Follow-up – how will you assess how well the countermeasures work and share the learning?
Most traditional businesses focus only on results and don’t pay attention to the methods. If they don’t get the desired results, they tend to replace the people. A better way is to focus on the methods rather than the results. Then if we don’t get the results we want, we can improve the method.
Design your experiments (use DOE)
Design of Experiments (DOE) is probably the single most powerful technique that you can use for problem solving, scientific inquiry, product and process development, refinement, and optimization. Most researchers seem to prefer to vary one factor at a time (OFAT) while keeping all the others constant. While OFAT seems scientific, it often cannot determine a true cause-effect relationship. In particular, OFAT is not capable of identifying interactions between factors. Nature works through interactions (think F = m*a), so experiments that ignore interactions are not useful. OFAT is subjective, not thorough, relies on luck, and takes too much time and money. Sir Ronal Fisher developed DOE to overcome the weaknesses of OFAT. DOE is objective and efficient, allowing you to maximize the information gained for the money spent. Modern software makes it easy to generate designs that fit the particulars of your experiment and analyze the results. Be sure to always randomize your run order to protect against unknown external factors and to repeat some trials to measure response variation (8 replicates is a good rule of thumb).
Capture usable knowledge in trade-off curves
The output of research is new usable knowledge (validated learning). Trade-off curves are the best way to transform data into usable knowledge and eliminate the waste of discarded knowledge. Good trade-off curves represent a lot of data clearly and show the limits of performance. Good trade-off curves also clarify what to do to improve. In particular, DOE models make good trade-off curves. Research teams should focus on creating new or shifting existing trade-off curves. Look for comments like “you can’t do that” or “that’s not possible.” Many times, there is tremendous value if you can figure out how to “do that.” If you cannot “do that” now, wait and later maybe you can. If someone has an idea that has been tested in the past, it is often worthwhile to run the experiment again because of the many changes that have occurred between then and now. Make sure you test to failure to expose the limits of your current understanding.
Work in Pasteur’s Quadrant
The late political scientist Donald E. Stokes divided research activity into four quadrants as shown below. He labeled the region where research advances basic science (the fundamental understanding of phenomena) as part of a quest to solve well-defined, use-inspired needs as “Pasteur’s Quadrant.” This quadrant is named for Louis Pasteur, a founder of the field of microbiology, who invented ways to prevent disease and food spoilage.
Typical methods for planning and tracking projects are not well suited to Pasteur’s Quadrant. Projects here involve fast iterations, so planning should be light and nimble. Progress can be assessed by tracking iterations to see if they are converging on goals, revealing dead ends, uncovering new applications, or identifying the need for unforeseen scientific advances. Insisting that a team steadily hit milestones established in initial plans can cause it to adhere to a path that no longer makes sense based on something the team has learned. Sometimes a setback or a failure is the most effective tool for discovery. If people working on a particular piece of a project experience a failure, it’s often because something they encountered surprised them. That’s to be expected in high-risk projects. When such events occur, the project leader has to let the team members press forward as long as they can see that the approach might ultimately work within project constraints, even if they deviate from the original course. That said, if it becomes clear that a given scientific approach won’t work or requires multiple miracles in a row, that particular effort should be shut down and the resources shifted to other approaches.
Use speed of learning as a sustainable competitive advantage
Steven Spear explains that the difference between good companies and great companies is their rate of learning. Learning happens through repeated problem-solving cycles. Toyota did not start out making high-quality cars at low cost. In 1965, Toyota’s productivity was about half that of GM. However, Toyota was learning how to improve quality and reduce cost at a rate about 3.5 times faster than GM so that by 1990, Toyota’s productivity was twice GM’s. Toyota and other fast learning organizations have turned their rate of learning into a sustainable competitive advantage. Remember that mistakes are OK. Risks and failures help us learn. We learn from failures because they reveal boundaries in our system’s current capability and horizons in our mind. We can minimize the risk of failure with speed and low-cost experimentation to develop ideas and gain clarity. Knowledge that is valuable now frequently is nearly useless in a month because we have already made the mistake it might have prevented. The late Allen Ward said, “Competitive advantage derives from discovering new principles, or new applications of basic principles, specific to your products and obtainable only from your experience.” If we can learn faster than our competition, we are certain to win!
Strive for continuous improvement
It is arrogant to believe that anything we have created cannot be improved, and it is pessimistic to believe that we are incapable of ever improving something that is flawed. If you are not improving, you are going backward (think the second law of thermodynamics). Focus on moving closer to the “ideal state” whenever you are debating the direction of your next continuous improvement activity. We are each responsible for improving our daily work processes and the systems we are part of. We must learn to continually assess the current state of our processes and pursue a better future state that will enhance value (or eliminate waste) and thus continuously improve.