A recent study found that some common combinations of emulsifiers, dyes and artificial sweeteners were associated with a higher risk of Type 2 diabetes beyond what could be explained by the individual food additives alone. The International Council of Beverages Associations (ICBA) called the study’s results “misleading” and confusing for consumers:

Decades of robust scientific evidence show these ingredients are safe. Suggesting that recipes—a combination of safe ingredients—are worrisome is simply absurd.

Safe ingredients on their own does not imply safe ingredients in combination. The ICBA comment illustrates a lack of understanding regarding interactions that I believe is due, in part, to the way we teach science in American schools. In particular, the one-factor-at-a-time (OFAT) experimental methodology most students learn is blind to interactions. In biological and other complex systems, interactions are not a rare nuisance; they are often the most important part of the story.

What is an Interaction?

An interaction occurs when the effect of one factor is dependent on the level of another factor. Think about baking a cake. The ideal baking time depends on the oven temperature. Thirty minutes might be perfect at 350°F, but it will be too short at 300°F and will burn the cake at 400°F.

The natural world is full of interactions. Newton’s Second Law is a classic example in physics. The force required to move an object is the product of its mass and the desired acceleration. The effect of mass on the system is entirely dependent on the level of acceleration. If you are not accelerating, the mass of the object has no impact on the net force. However, as acceleration increases, the cost of that mass in terms of required force scales multiplicatively. Conversely, for a fixed amount of force, the resulting acceleration is inversely dependent on the mass. A heavier object resists the effect of the force more than a lighter one. Ohm’s Law and the Ideal Gas Law are additional classic examples of interactions in physics.

In chemistry, the Haber-Bosch process creates ammonia for fertilizer and is essential for feeding the world’s population. It relies on an iron catalyst interacting with nitrogen and hydrogen gases under high pressure and temperature. Without the interaction with iron, the reaction is too slow to be practical.

In medicine, a synergistic interaction occurs when the combined effect of two drugs is greater than the sum of their individual effects, like aspirin and caffeine for pain relief. An antagonistic interaction occurs when the interaction is harmful, like alcohol and antidepressants. Cancer patients are often treated with a “cocktail” of therapies, such as chemotherapy and radiation, or multiple chemotherapy agents at once because the combination treats cancer more effectively than any of the treatments alone.

Our genes don’t operate in a vacuum, either. Their expression is profoundly influenced by the environment, and this interplay determines traits and disease risk. In a famous study of adoptees, those with a high genetic risk for schizophrenia (their biological mothers had it) who were raised in a disturbed family environment had a 36.8% chance of developing the disorder. However, those with the same high genetic risk who were raised in a healthy environment had only a 5.8% chance, nearly the same as the low-risk group.

Ecological interactions between organisms and their environment are abundant. Bees get nectar from flowers while the flowers get pollinated, and fungi form mycorrhizal networks with tree roots, helping the tree absorb nutrients in exchange for sugars.

From the molecular level to entire ecosystems, nature appears to work primarily through interactions. So why did the ICBA claim interactions between food additives was absurd?

One-Factor-At-A-Time (OFAT)

Think back to your first, and perhaps only, exposure to the “scientific method.” Most likely you were taught something like the following:

1. Ask a Question

2. Form a Hypothesis

3. Conduct an Experiment

4. Analyze the Data

5. Draw a Conclusion

When it came time to conduct the experiment, you were taught to systematically hold every conceivable condition constant while altering just a single factor. Any subsequent change observed in the outcome could then be attributed to that lone, isolated change. This one-factor-at-a-time (OFAT) approach for controlling variables is deeply embedded in how science is defined and taught in American schools. Nearly all science fair guidelines, including those from NASA, explicitly state the requirement to test only one factor at a time.

While not completely wrong, OFAT experimentation suffers from serious weaknesses: it completely ignores factor interactions, it isn’t efficient or thorough, and it is highly subjective. The problems with the OFAT method were identified over 100 years ago by Sir Ronald Fisher.

Fisher’s Insight

In 1919, Ronald Fisher was hired by the Rothamsted Experimental Station in Hertfordshire, England, to make sense of the vast amount of data from their Classical Field Experiments, some of which had been running continuously since 1843. Fisher recognized that the nature of agricultural research imposed constraints that made OFAT experimental methods deeply problematic. The primary limitation was time. An experiment testing the effect of a new fertilizer on crop yield took an entire growing season to complete. Testing several different fertilizers and crop varieties one by one would take many years, a timescale that was both scientifically and economically untenable.

Compounding the issue of time was the problem of uncontrollable variation. An agricultural field is a heterogeneous system. Slight differences in soil, drainage, sun exposure, and pest infestation could introduce variability or “noise” into the results. This background noise could easily be larger than the actual effect of the treatment being studied, making it impossible to determine if any improvement was due to the treatment or simply due to that plot being on a slightly better patch of soil.

The need to get reliable answers from a limited number of trials in a noisy, complex environment created the imperative for a new science of experimental design. As Fisher delved into the Rothamsted data and began designing new experiments, he came to a conclusion that placed him in direct opposition to the established scientific wisdom regarding OFAT. He articulated this conclusion in his 1926 paper published in the Journal of the Ministry of Agriculture:

No aphorism is more frequently repeated in connection with field trials, than that we must ask Nature few questions, or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken. Nature, he suggests, will best respond to a logical and carefully thought out questionnaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed.

Fisher’s critique was not a minor quibble about technique. He was proposing a comprehensive dismantling of the logic that underpinned the traditional OFAT approach. His argument rested on three interconnected flaws he identified in the OFAT methodology.

1. OFAT Cannot Identify Interactions

Fisher’s most profound critique was OFAT’s fundamental inability to detect or measure interactions. OFAT varies only one factor at a time and assumes that interactions do not exist. This is a fundamental structural limitation. There is no systematic way to find an interaction with OFAT because the data required simply doesn’t exist.

Going back to the cake baking example, the problem is to produce a good tasting cake. The chef decides to experiment with oven temperature using the OFAT approach, holding the baking time fixed at 30 minutes along with all other known factors (type of oven, type of baking dish, altitude, etc.). The cake was raw at 300°F and burnt at 400°F. The chef decides the best temperature to bake a cake is 350°F. The effects of the other factors are not measured. There is no data showing how temperature and time interact, so there is no way to see that a good cake could be obtained by extending the baking time to 60 minutes at 300°F. Fisher realized that in a world governed by complex interdependencies, a method that cannot see them is not just limited, it is a recipe for error.

2. OFAT is Inefficient

The most practical flaw Fisher identified was OFAT’s inefficiency. In an OFAT experiment, each trial provides information about only one factor. Fisher argued this was a waste of resources in the context of agricultural trials where each run took a full year. His proposed alternative, what became known as the Design of Experiments or DOE, allows a researcher to investigate multiple factors simultaneously.

The key insight is that in a DOE, every single trial provides information on every factor being studied. For example, in a simple experiment with two fertilizers and two crop varieties, a factorial design would have four treatment groups:

To estimate the main effect of the fertilizer, one compares the average yield of the two Fertilizer 1 experiments (group 1 and 2) with the average yield of the two Fertilizer 2 experiments (group 3 and 4). Crucially, the entire experiment (all four groupings) contribute to this estimate. Similarly, all four groups contribute to the estimate of the effect of crop variety. Fisher argued one could learn as much about many factors as one could learn about a single factor in the same number of trials, dramatically increasing the efficiency and power of the experiment.

3. OFAT is Subjective

Because the OFAT method explores the potential experimental space one dimension at a time, it can easily lead the researcher to a “false optimum.” The result of an OFAT optimization process is entirely dependent on the initial starting conditions and the order in which the factors are tested. In other words, the results are subjective and rely on luck.

Think about climbing a mountain in thick fog. The climber starts at base camp and walks in one direction (say North) until the altitude no longer increases. This is the local North-South peak. From that point, they walk in another direction while holding the first direction constant (say East) until the altitude no longer increases. The climber now believes they have reached the summit. However, the true summit of the mountain might lie to the Northeast, a direction they were forbidden from traveling. By only moving in cardinal directions (one variable at a time), they have trapped themselves on a lower ridge, a false optimum. Had the climber started at a different base camp or chosen to explore East-West first before North-South, they might have ended up on a different ridge entirely, potentially closer to or further from the true peak.

Fisher found this path-dependency “very unsatisfactory.” A reliable scientific method cannot produce different optimal solutions based on the arbitrary order of operations. It must be able to survey the entire landscape to find the true peak. The OFAT method, by its very nature, explores only a narrow, cross-shaped path through the experimental space. The unthorough investigation leaves vast areas in between, where the true optimum often lies, completely unexplored.

Consequences of OFAT Thinking

The ICBA’s mistake is an unavoidable consequence of OFAT thinking. An experimental system that is structurally unable to detect interactions, too inefficient to find them even if they exist, and entirely reliant on arbitrary starting conditions will naturally dismiss any claims of complexity. This is precisely why the combined effects of food additives were declared “absurd.”

Fortunately, there is a better way. Fischer’s work spurred him to develop new methods of designing experiments that could simultaneously provide answers to many questions, most of which could not even have been asked using one question at a time. Fischer’s methods are commonly referred to as the Design of Experiments or DOE. DOE transformed statistics from a descriptive tool for summarizing data into a prescriptive and inferential science for generating new knowledge. The experimental design is of paramount importance because no amount of information can exceed the quantity supplied by the data. If the data doesn’t exist, the effect cannot be measured.

The Design of Experiments (DOE) framework enables you to move past subjective, inefficient, and blind experimentation, accelerating your innovation. If you would like to learn more about how to incorporate DOE methods into your experiments, please reach out to me at ken@objexp.com or see objectiveexperiments.com.

Introduction:

In 1999, my boss asked me a simple question: "Can we make polycrystalline diamond cutters (PDC) any better, or is 'diamond just diamond'?" I was working in R&D at US Synthetic developing PDC inserts for oil and gas drilling, and I knew a "no" wasn't the right answer. So, I took a leap of faith and said, "I think we can make it 50% better." Little did I know that conversation would lead to a 25-year journey of relentless experimentation and a thousand-fold (three orders of magnitude) improvement in PDC wear resistance!

The Challenge:

PDC drill bits were primarily used for soft rock formations. Harder rocks required roller cone bits with tungsten carbide inserts. The inherent brittleness of diamond limited its application, even though diamond is the hardest material known. Our goal was to push the boundaries and make PDC suitable for all rock types.

Discovering the Future:

We did not have an equivalent of Moore’s Law to show us that exponential growth in performance over decades was possible. Our innovations weren't "Eureka!" moments. Instead, improving the performance of PDC was a gradual, evolutionary process, fueled by constant learning and adaptation resulting from the following:

• Massive Experimentation: We tested everything, embracing failure as a learning tool. Our journey involved over 30,000 experiments and 200,000 lab-broken inserts. We dedicated equipment and personnel to a prototype factory specifically for the purpose of building inserts we could test. 

• Fast Learning: Innovation isn't just about testing; it's about speed. Speed of learning. We had to develop systems to accelerate our learning process, which proved crucial to our success. 

• Teamwork: This was a collective effort, not a solo pursuit. Collaboration was essential to improve the quality of our experiments and to learn from our discoveries. 

• Freedom to Explore: We had the space to challenge conventional wisdom, even when it meant taking risks. We were told not to experiment with pressure, but we did anyway. It led to equipment failures and increased costs, but also to significant breakthroughs. We had to innovate not only the PDC but also the tools we used to make it. We delved into the details of leaching, starting with hot plates and Pyrex dishes! We struggled to sinter fine-grain diamond powders until we learned the importance of surface chemistry and adsorbed gasses. Each area provided crucial insights.

This quote perfectly encapsulates our journey. We were constantly combining and recombining ideas, learning from every experiment, and gradually refining our processes. The improvement was not just incremental; it was transformative. Our massive industrial effort to push forward the technological frontier improved PDC wear resistance by three orders of magnitude!  

The Impact:

Today, PDC bits drill successfully in all rock formations, even hard, hot granite for geothermal applications. The advances we drove in PDC performance have led to incredible gains in efficiency and productivity. These innovations played a key role in the U.S. becoming the largest oil producer in the history of the world.

Conclusion:

I'm incredibly grateful I didn't simply say, "Diamond is just diamond," in 1999. Our journey has shown me the power of relentless curiosity, the importance of teamwork, and the rewards of pushing boundaries. What are your experiences with innovation and exponential performance improvements? Share your stories in the comments below!

Disagreement is not disrespect

Disagreements will occur as a natural consequence of the research process. In unhealthy organizations, the resultant conflict leads to mutual distrust and suspicion and causes significant knowledge waste and unnecessary friction. In healthy organizations, conflict drives information exchange and helps build consensus. Keep in mind that it is difficult to learn anything new by only listening to those who agree with you. Speak with data so that disagreements do not become personal. Better yet, stop debating, get out of the office or conference room, and run the experiment together. The feeling of being in a great culture is not smoothness or lack of conflict – it’s the feeling of solving hard problems with people you admire.

Be a “learner,” not a “knower”

Innovation comes at the intersection of ideas, and good ideas can come from anywhere. A “learner” is someone humble enough to abandon ownership of their ideas, seek input from those they don’t always agree with, and then run the experiment. Carol Dweck calls this a growth mindset. Learners tend to embrace challenges, persevere in the face of adversity, accept and learn from failure, focus on process rather than outcome, and see abilities as skills which are developed through effort. They are more likely to see effort as something that propels learning and to see feedback and failure as opportunities to build new skills. A “knower,” on the other hand, is heavily invested in their own ideas, rarely seeks input from others, and believes their status and credentials ensure their authority. If you believe that your qualities are unchangeable — what Dweck calls a fixed mindset — you will want to prove yourself correct over and over rather than learning from your mistakes. To be a good problem solver, you must adopt the mindset of a learner. Moreover, skilled learners avoid the temptation to solve other people’s problems for them. Helping people solve challenging problems on their own represents one of the highest forms of respect.

Beware of confirmation bias

Confirmation bias is the tendency to pay attention to data that supports your view and ignore everything else. Confirmation bias is a powerful human tendency, and we must always be on guard against its influence. We have to actively recognize our biases and temporarily suspend them in our mind as we take in new information and external data so we can be as objective as possible. It is a fallacy to think we don’t need to do this actively, that we’ll naturally do it passively. Recognize that you don't know everything and that your beliefs may be wrong. Be open to learning new things and changing your mind. You stand a better chance of overcoming bias if you tackle an innovative project as part of a team rather than trying to do it alone. Working as part of a team exposes you to people who have different beliefs than you and can help you to see the issue from a different perspective.

State your hypothesis

Richard Feynman describes the scientific method as

• Observation – the gathering and recording of data about the natural world

• Reason – the ability to think, understand, and draw conclusions

• Experiment – a test carried out to see how something works

For an experiment to be scientific, it must be possible that the hypothesis will be refuted. It is extremely important that you write down what you expect to happen before you conduct an experiment. If there’s no written prediction, then your brain can trick you into thinking that the results are what you expected (confirmation bias). Learning most often occurs when there is a difference between what we expected to happen and what actually happened.

It is important to remember that uncertainty is normal in research. We cannot predict the future, nor can we think our way to the future. We can find the path to the future only by repeated application of the scientific method. We must move beyond testing to distinguish good from bad. We must structure our experiments to distinguish understood from not understood. Also, be aware that an unexpected observation made during an experiment cannot be used to validate itself. The new observation must now be tested on its own.

Avoid jumping to solutions

When solving a problem, our natural human tendency is to “jump to solutions.” We assume we know what the problem is without seeing what is actually happening, we assume we know how to fix a problem without finding out what is causing it, and we assume we know what is causing the problem without confirming it. To be better at problem solving, we need to help ourselves see what is actually happening and what we actually know. A powerful tool to prevent jumping to a solution is to follow a problem-solving method. Following a good method forces you to truly understand the problem before taking any action. John Shook devised the following method based on his observations of Toyota leaders:

• Review the background – what are you talking about and why?

• Grasp the current condition and state the problem – where do things stand now?

• State the goal or target – what specific outcome is required?

• Analyze the root cause – why does the problem exist?

• Present countermeasures – what do you propose and why?

• State an implementation plan – how will you implement the countermeasures?

• Follow-up – how will you assess how well the countermeasures work and share the learning?

Most traditional businesses focus only on results and don’t pay attention to the methods. If they don’t get the desired results, they tend to replace the people. A better way is to focus on the methods rather than the results. Then if we don’t get the results we want, we can improve the method.

Design your experiments (use DOE)

Design of Experiments (DOE) is probably the single most powerful technique that you can use for problem solving, scientific inquiry, product and process development, refinement, and optimization. Most researchers seem to prefer to vary one factor at a time (OFAT) while keeping all the others constant. While OFAT seems scientific, it often cannot determine a true cause-effect relationship. In particular, OFAT is not capable of identifying interactions between factors. Nature works through interactions (think F = m*a), so experiments that ignore interactions are not useful. OFAT is subjective, not thorough, relies on luck, and takes too much time and money. Sir Ronal Fisher developed DOE to overcome the weaknesses of OFAT. DOE is objective and efficient, allowing you to maximize the information gained for the money spent. Modern software makes it easy to generate designs that fit the particulars of your experiment and analyze the results. Be sure to always randomize your run order to protect against unknown external factors and to repeat some trials to measure response variation (8 replicates is a good rule of thumb).

Capture usable knowledge in trade-off curves

The output of research is new usable knowledge (validated learning). Trade-off curves are the best way to transform data into usable knowledge and eliminate the waste of discarded knowledge. Good trade-off curves represent a lot of data clearly and show the limits of performance. Good trade-off curves also clarify what to do to improve. In particular, DOE models make good trade-off curves. Research teams should focus on creating new or shifting existing trade-off curves. Look for comments like “you can’t do that” or “that’s not possible.” Many times, there is tremendous value if you can figure out how to “do that.” If you cannot “do that” now, wait and later maybe you can. If someone has an idea that has been tested in the past, it is often worthwhile to run the experiment again because of the many changes that have occurred between then and now. Make sure you test to failure to expose the limits of your current understanding.

Work in Pasteur’s Quadrant

The late political scientist Donald E. Stokes divided research activity into four quadrants as shown below. He labeled the region where research advances basic science (the fundamental understanding of phenomena) as part of a quest to solve well-defined, use-inspired needs as “Pasteur’s Quadrant.” This quadrant is named for Louis Pasteur, a founder of the field of microbiology, who invented ways to prevent disease and food spoilage.

Typical methods for planning and tracking projects are not well suited to Pasteur’s Quadrant. Projects here involve fast iterations, so planning should be light and nimble. Progress can be assessed by tracking iterations to see if they are converging on goals, revealing dead ends, uncovering new applications, or identifying the need for unforeseen scientific advances. Insisting that a team steadily hit milestones established in initial plans can cause it to adhere to a path that no longer makes sense based on something the team has learned. Sometimes a setback or a failure is the most effective tool for discovery. If people working on a particular piece of a project experience a failure, it’s often because something they encountered surprised them. That’s to be expected in high-risk projects. When such events occur, the project leader has to let the team members press forward as long as they can see that the approach might ultimately work within project constraints, even if they deviate from the original course. That said, if it becomes clear that a given scientific approach won’t work or requires multiple miracles in a row, that particular effort should be shut down and the resources shifted to other approaches.

Use speed of learning as a sustainable competitive advantage

Steven Spear explains that the difference between good companies and great companies is their rate of learning. Learning happens through repeated problem-solving cycles. Toyota did not start out making high-quality cars at low cost. In 1965, Toyota’s productivity was about half that of GM. However, Toyota was learning how to improve quality and reduce cost at a rate about 3.5 times faster than GM so that by 1990, Toyota’s productivity was twice GM’s. Toyota and other fast learning organizations have turned their rate of learning into a sustainable competitive advantage. Remember that mistakes are OK. Risks and failures help us learn. We learn from failures because they reveal boundaries in our system’s current capability and horizons in our mind. We can minimize the risk of failure with speed and low-cost experimentation to develop ideas and gain clarity. Knowledge that is valuable now frequently is nearly useless in a month because we have already made the mistake it might have prevented. The late Allen Ward said, “Competitive advantage derives from discovering new principles, or new applications of basic principles, specific to your products and obtainable only from your experience.” If we can learn faster than our competition, we are certain to win!

Strive for continuous improvement

It is arrogant to believe that anything we have created cannot be improved, and it is pessimistic to believe that we are incapable of ever improving something that is flawed. If you are not improving, you are going backward (think the second law of thermodynamics). Focus on moving closer to the “ideal state” whenever you are debating the direction of your next continuous improvement activity. We are each responsible for improving our daily work processes and the systems we are part of. We must learn to continually assess the current state of our processes and pursue a better future state that will enhance value (or eliminate waste) and thus continuously improve.

One of the displays was titled, "How to Make Dreams into Reality." It described a five-step method for designing and engineering an aircraft. The five steps are as follows:

• ASK: What need is met by this product? Who will use it?

• IMAGINE: Draw or write about your ideas. Let your imagination soar. Develop several possible solutions.

• PLAN: Pick the best option. Get help and form a team. Work together to create a plan or blueprint.

• CREATE: Build a mockup or prototype to test.

• IMPROVE: Test your prototype. Discover how it succeeds or fails. Make improvements and retest.

This appears to be another example of the scientific method applied iteratively to a problem. Other similar methods include Deming's PDCA cycle (Plan-Do-Check-Adjust), Boyd's OODA loop (Observe-Orient-Decide-Act), and Shook's A3 problem solving method. Using a good method is critical for effective problem solving, something I hope to write more about in the future.

Exercise 2.13. Using a standard deck of 52 cards, you are dealt 13. What is the probability of getting six spades?

Binomial Distribution

My first attempt to solve the problem utilized the binomial distribution. The binomial distribution is used when there are exactly two mutually exclusive outcomes of a trial (like drawing a spade or not-spade). These outcomes are labeled a "success" or "failure". The binomial distribution gives the probability of y successes out of a n possible trials where the fixed probability of an individual success is p and of a failure is q = 1 - p as follows:

Obviously, the assumption of a fixed probability of success is not correct for the card drawing problem because the probability of drawing a spade changes with each card drawn. However, for a population N much larger than the sample n (in this case N = 52 and n = 13), the binomial distribution can be a good approximation. I wrote the following Python script to calculate the probability:

from scipy.special import factorial

###########################################
def binomial_probability (yy, nn, pp, qq):

    p_q = (pp**yy) * (qq**(nn-yy))
    prob = p_q*(factorial(nn, exact=True)/(factorial(yy, exact=True)*factorial(nn-yy, exact=True)))
    return (prob)

#########################################
# Exercise 2.13  - Binomial distribution

# As a first approximation, assume the binomial distribution

p = 13./52.
q = 39./52.
n = 13
y = 6

print('Pr(six spades in hand of 13) = ', binomial_probability(y, n, p, q))

The probability of getting six spades in the first 13 cards drawn from a 52-card deck is 0.056 or about 1/18 as determined by the Binomial Distribution. However, given the fact that the probability of success is not fixed, I wanted to find a more accurate result.

Monte-Carlo Simulation

In this approach, I used a Monte-Carlo method to simulate drawing 13 cards from a 52-card deck. The "deck" consisted of 13 one's representing spades and 39 zero's representing not-spades. The Python code below randomly selects 13 cards from the deck (a "hand") and keeps track of how many times there were exactly six spades in the hand.

import numpy as np

# Exercise 2.13 - Monte Carlo method

# fill the card deck with zeros (non-spades)
deck = np.zeros(52, dtype=np.int32)

# change the first 13 values to 1 (spades)
deck[:13] = 1

# randomly sample 13 cards from the deck and make a new list
hand = [] # list of cards in one hand
num_runs = 1000000 # number if simulated hand draws
i = 0

# count the number of times six spades are present in a hand
six_spades = 0 
while i <= num_runs:
    # draw a hand of 13 cards randomly with no replacement
    hand = np.random.choice(deck, size=13, replace = False)
    # since spades == 1 and non-spades == 0, sum gives the number
    # of spades    
    spades = np.sum(hand)  
    if spades == 6:
        six_spades = six_spades + 1
    i = i + 1
    
print(six_spades)
print('Pr(six spades in hand of thirteen) = ', six_spades/num_runs)
    

The probability of getting six spades in the first 13 cards drawn from a 52-card deck is 0.042 or about 1/24 as determined by averaging the results of six 1,000,000 draw simulations. This result is about 1/3 smaller than the binomial result, which is not surprising since the probabilities get smaller initially as more cards are drawn. The Monte-Carlo result was closer to true, but I believed there must be a way to actually calculate the exact probability.

Calculating the Probability of Each Hand

The actual probability of drawing a spade or not-spade changes for every card drawn and whether or not a spade was drawn. For example, the probability of getting a spade in the first draw is 13/52, and the probability of getting a not-spade is 39/52. Assume the first card drawn is not a spade. Then the probability of getting a spade with the second draw is 13/51, and the probability of getting a not-spade is 38/51.

There are 1,716 different ways to get six spades in a hand of 13. Each arrangement has it's own probability, which is the product of the individual probabilities for each card drawn in sequence. The Python code below calculates the actual probabilities for each possible combination.

import numpy as np
from itertools import product

# Exercise 2.13 - Find probability of each combination of hands

# Create a list of hands covering every combination of 6 spades 
# in a 13 card hand. Values == 1 are spades,
# and values == 0 are not spades.

cards = [0, 1]
hands = product(cards, repeat=13)
# turn hands into a list
all_hands = list(hands)

# just keep the hands with six spades
six_spades = []
for hand in all_hands:
    if np.sum(hand) == 6:
        six_spades.append(hand)

# now find the probability of each hand
prob_hand = []
for hand in six_spades: # iterate over each hand
    probability = 1. # reset probability of drawing the hand to 1
    card_count = 52 # reset the number of cards to a full deck
    spade_count = 13 # reset the number of spades to 13
    non_count = 39 # reset the number of non-spades to 39
    for card in hand: # iterate over the cards in each hand
# if not-spade, calculate probability of drawing that card
        if card == 0: 
            probability = probability * non_count/card_count
            non_count = non_count - 1
            card_count = card_count - 1

# if spade, calculate probability of drawing that card
        else: 
            probability = probability * spade_count/card_count
            spade_count = spade_count - 1
            card_count = card_count - 1
    prob_hand.append(probability) # create a list of probabilities for each individual hand

# the probability of drawing 6 spades is the sum of the 
# probabilities of each hand combination

print('Possible combinations = ', len(prob_hand))
print('Pr(six spades in thirteen cards) = ', np.sum(prob_hand))
print('Probability is one out of ', 1./np.sum(prob_hand))

The true probability of getting six spades in the first 13 cards drawn from a 52-card deck is 0.0416 or about 1/24 . This is approximately equal to the average result from the Monte-Carlo simulation of 1,000,000 draws.

Hypergeometric Distribution

Ben Kendall pointed out to me that this problem could be solved using a hypergeometric distribution probability calculator. The hypergeometric distribution is a discrete probability distribution that describes the probability of y successes in 𝑛 draws, without replacement, from a finite population of size 𝑁 that contains exactly 𝐾 objects with that feature, wherein each draw is either a success or a failure. The classic application of the hypergeometric distribution is sampling without replacement, which is a good description of Exercise 2.13. The results using the hypergeometric distribution are the same as those from the probability calculations above. The hypergeometric distribution can also provide cumulative probabilities.

Probabilities of Drawing y Spades

The table below summarizes the probabilities of drawing one through 13 spades using each of the above three methods.

Note that the Monte-Carlo method fails to draw any hands with 11 to 13 spades. This is because I only simulated 1,000,000 draws. The true odds of drawing 11 spades is about 1/11,000,000, and the odds for 12 and 13 are even lower. Thus I'd have to simulate many more draws (over about 635 million in the case of 13 spades) to get an accurate estimate for these lower probability events.

Note also that the textbook lists the answer as 0.13. I believe the values I calculated above are correct. Perhaps the book mistakenly shows the result for getting five spades, which is equal to 0.13 when calculated using the Binomial Distribution.

A common problem in manufacturing is to determine if a process change affects the quality of a product. The problem then is to compare two populations experimentally and decide whether differences that are observed are genuine or merely due to chance. This raises an important question about the choice of a relevant reference distribution. A recent experience I had with a colleague serves as a useful illustration.

A change was made to the production of a raw material used in our manufacturing process. This raw material is used to process our finished products and is not a component of the product itself. Extensive testing showed that the material properties of the raw material were not affected, so the decision was made to implement the change. As a precaution, we measured a substitute quality characteristic of three finished products per batch for 42 weeks prior to the change and 6 weeks after the change. 

A colleague chose to conduct a hypothesis test on the mean values of the substitute quality characteristic before and after the change. A plot of the data is shown in Figure 1. The mean value before the process change (Process A) was 1456 [1449-1464, 95%, N=4230]. The mean value after the process change (Process B) was 1522 [1473-1527, 95%, N=90]. The difference in means was 66 with a t-statistic of 2.6. The probability of a t-statistic of 2.6 or larger is 0.0048 (p = 0.0048). Thus my colleague rejected the null hypothesis that the means were not different, and concluded that the difference between Process A and B was genuine and not due merely to chance. However, because of the considerable variability in the measurements before the change, my colleague wondered whether this analysis method was valid. 

In order for the hypothesis test as conducted to be valid, the measurements from Process A (the reference set) should be normally, identically, and independently distributed relative to measurements from Process B. If these assumptions are not true, then the data from Process A is an inappropriate reference set.

An elegant method of constructing a relevant reference set is described in Statistics for Experimenters by Box, Hunter and Hunter. They propose using the 4230 measurements made before the change to determine how often in the past had differences at least as great as 66 (the difference in means between Process A and B) occurred between averages of successive groups of 90 (the number of parts measured for Process B). The distribution of these 4141 differences between averages of adjacent sets of 90 observations is shown in Figure 2. These differences provide a relevant reference set with which the observed difference of 66 (red line in Figure 2) may be compared. In this case, a difference of 66 or greater was observed 803 times.

We can say that, in relation to this reference set, the observed difference was statistically significant at the 803/4141 = 0.19 level of probability (p = 0.19). In other words, the probability of observing a difference of 66 or larger merely by chance is 19%, which provides little evidence against the null hypothesis that the observed difference is part of the reference set. Thus the difference observed by my colleague was not large enough to conclude that there was a change in the product quality. 

Unlike the original hypothesis test, this approach does not require the data to be independent or normal. It only assumes that whatever mechanisms gave rise to the original observations (Process A) are operating after the change (Process B). As such, it is a useful technique for comparing two populations.

But why are the probabilities for the null hypothesis so different between these two methods (p = 0.0048 compared to p = 0.19)? Figure 3 shows a plot of each measurement versus the adjacent previous measurement. The plot shows a clear auto-correlation of adjacent observations in the Process A data. The positive autocorrelation in these data produces an increase in the standard deviation. Thus the reference distribution in Figure 2 has a larger spread than the corresponding scaled t distribution in Figure 1. The student’s t test gives the wrong answer because it assumes that the errors are independently distributed, which, as shown in Figure 3, they are not. 

Because of influences such as serial correlation, the t test based on normal, independent, and identically distributed assumptions will not be valid. A very useful method for resolving this problem is to generate an external reference distribution from the past data as described above.

If you do not have access to past data, you might be able to use a randomization distribution from the results of a randomized experiment to supply a relevant reference set. This method was originally proposed by Ronald Fisher in his 1935 book The Design of Experiments and further outlined in Statistics for Experimenters.

The table below lists my Strava stats for the race alongside the 50-mile 1st and 2nd place winners, Zach Calton and Danny Van Wagoner, respectively. I included the 2nd place winner because his power data was recorded with a power meter. One interesting observation is that my Energy Output was nearly the same as Zach's. Zach's Estimated Average Power was about double mine, which makes sense considering Power = Energy/Time, and the winner finished the race in half my time. Another interesting observation is that the Calories number is 3.7 to 5.6 times greater than the Energy Output, even though they are both calculations of energy.

Strava's Estimated Average Power is either the measured power at the cranks via a power meter or an estimate based on your weight, speed, and elevation change. The data for me and Zach are both estimates, while Danny's data was measured by a power meter. Strava then uses this data to estimate the mechanical energy needed to move through the course (Energy Output) or, in Danny's case, the actual Total Work done. Danny's value for Total Work is the benchmark for this analysis since it is based on actual measurements with a power meter at the cranks. I will try to calculate the Energy Output/Total Power for True Grit and compare it to the Strava values.

Strava's Calories is a measure of the energy an athlete burns during an activity. Some of that energy is used to move the cranks, some is lost to inefficiency in our bodies converting fuel to muscle motion, and some is lost to the environment as heat (convection, sweat, and respiration). Therefore, Calories should be higher than Energy Output/Total Work.

Energy Output/Total Work

Consider the bike and the rider as an isolated system. The total energy contained within the system boundary can be given the symbol E and is made up of the kinetic energy (KE), the potential energy (PE), and the internal energy U:

At the start of the race, the kinetic energy is zero because the system is at rest. The potential energy is also zero if we assume the start line is the reference plane. The only energy in the system is the internal energy, i.e. chemical energy stored in our bodies and the food we carry with us to eat.  We are interested in the amount of energy that crosses our system boundary as we convert our internal energy into motion around the course.

It is useful to define work as an energy transfer across the system boundary in an organized form such that its sole use could be the raising of a weight. In mathematical terms, the amount of work W done in moving through a differential distance ds is

where the center dot denotes the dot product. The work done over a finite path between points a and b is

where F is the force in the direction s. Note that unless the path is specified from the initial to the final state of the system, we are not able to calculate the work done. Therefore, work is an inexact differential quantity in mathematics. The value of work depends on not only the initial and final states of the system but also the path followed. 

Energy that crosses the system boundary because of a temperature difference falls into a separate category. This energy transfer is not work, since we cannot envision a way in which its sole effect is the raising of a weight. We thus define this disorganized form of energy that crosses the system boundary because of a temperature difference between the system and its surroundings as heat transfer. 

Given these two categories, we can think of all the sources of energy transfer relevant to our system:

1. Gravity Work – the energy needed to overcome the force of gravity as we traversed the course

2. Drag Work – the energy needed to overcome wind resistance as we moved around the course

3. Rolling Resistance Work - the energy needed to overcome the force resisting the motion of our wheels rolling over the ground and the many obstacles on the True Grit course

4. Friction Work – the energy needed to overcome friction between moving components of the bike (wheel axle bearings, crank bearings, pedal bearings, chain links, etc.) 

5. Heat Loss – the energy we expend keeping our body warm, sweating, and respirating

6. Efficiency Loss - the energy we waste converting fuel in our bodies to muscular motion at the cranks

The Total Work for this analysis is just the work delivered at the crank (1-4 above). It neglects heat loss and efficiency losses (5 and 6).

1. Gravity Work

The work done by my system on the surroundings to overcome gravity can be represented by

The force due to gravity is the system mass (m) times gravitational acceleration (g), and is not a function of position on the course (neglecting the minor changes in g with elevation). We only need to consider the distance moved vertically since gravity only acts in the vertical direction. Thus the integral of ds is the vertical distance traveled, h. We do not consider the work done by gravity on our system during the descent because this energy does not transfer across the system boundary (we cannot absorb the energy during the descent for use later).

The image below shows the race elevation change from Strava. My total elevation gain was h = 1553 m (5095 ft). My mass, including the bike and my pack, is about 90 kg. Thus the energy I needed to overcome gravity was 1371 kJ. This value represents only about 15% of the estimated energy consumed during the race.

If I assume Zach and Danny have a mass of 90 kg, their gravity work is 1321 kJ and 1291 kJ, respectively.

2. Drag Work

The work done by my system on the surroundings to overcome drag can be represented by

Aerodynamic drag is the resultant force due to pressure and shear stress on a body’s surface as it moves through the air. Drag due to pressure is called form drag because it depends on the shape of the body. Drag due to shear stress is called skin friction drag because it depends primarily on the amount of surface in contact with the fluid. In general, drag force can be represented by

where rho is the air density, V is relative velocity between the body and the fluid (air), CD is the drag coefficient, and A is a representative area depending on the type of drag being considered.

Let's model our system as a flat plate oriented perpendicular to the approaching airflow. The drag on a normal plate is due to pressure only (form drag) and is insensitive to Reynolds number. If we assume the plate width is equal to our shoulder width (about 0.5 m), and the plate height is equal to our shoulder height (about 1.5 m), we can estimate the drag coefficient to be about 1.1.

The velocity V is a function of position on the course, as shown below from Strava.

Most of the high velocity periods were experienced during the descents where the drag work may have been completely overcome by the effect of gravity. Most of the climbing was done at low velocity where the drag work would be low. Thus, most of the drag work would be experienced on the relatively flat sections of the course where our velocity might be close to the average. If we assume the velocity is constant for the entire course, the drag work equation reduces to

where L is the length of the course and Vavg is our average velocity. Now I have to make a few more assumptions to do the calculation.

• Assume the average air temperature was 20 C and the elevation was 1000 m (air density = 1.112 kg/m^3)

• Assume there is no skin drag from the bike, and all the drag comes from form drag

• Assume the wind velocity is zero (conditions were calm during the race)

The resultant drag force for me is about 5 N and the drag work is about 347kJ. The drag work for Zach and Danny is about 1304 kJ and 1224 kJ, respectively. Their drag work is four times higher than mine due to the velocity squared term.  

3. Rolling Resistance Work

The force needed to overcome rolling resistance is given by

where CRF is the coefficient of rolling friction. CRF is a function of both the force (rider mass) and the velocity. Using the data published by bicyclerollingresistance.com for MTB tires rolling on a smooth steel drum, I estimate CRF to be between about 0.0034 and 0.012 for each tire. This is the range of CRF values for all mountain bike tires tested at 25 psi. Thus the rolling resistance force for both tires is between about 3 N and 10.6 N, assuming that one-half of my mass is carried by each tire. Multiplying by the distance traveled gives Rolling Resistance Work of between 208 kJ and 735 kJ.

4. Frictional Work

A simple model is to assume 5% of crank work is required to overcome friction. This seems like a reasonable assumption.

Adding it all up...

The table below shows the calculated total work for each athlete from the four work components listed above. I also calculated the average power as

Danny's measured Total Work of 3000 kJ is in between my calculated work estimates of 2859 and 3412 kJ. Thus, my calculations do a reasonably good job estimating the actual work. Strava's estimate for my Total Work is also in between the calculated ranges. Zach's calculated work is higher than Strava's estimate. Perhaps Strava doesn't do a good job estimating work for elite athletes? The calculated power values are also in the range for both me and Danny and high for Zach. Overall I think my calculations are reasonably accurate and capture the main components of the work done.

Back to the original question of how do I compare to the winners. It appears that the major difference is the form drag since the work scales as the velocity squared. It would also seem reasonable to assume the winners have lower rolling resistance. For example, Danny was riding a hard-tail with race tires, and I was riding a full-suspension bike with wide knobbys.

Now look at the Strava data for Calories, or the amount of energy the athlete had to burn to deliver the work needed to move around the course. All of this extra energy was lost to the environment through heat transfer, sweat, and respiration. This gives me a greater appreciation for the effort it takes an elite athlete to win a race like True Grit.

1. Your opponent gets to move first.

2. You must give up some of your pieces.

Would you bet $1,000 that you could win? Would you even be interested in playing? 

What if you could move twice for every move your opponent made? Would the advantage of speed make up for your disadvantages? At US Synthetic, we experienced the power of speed when we reduced our learning cycle time from 45 days to 4 days. 

Rapid Learning Cycles at US Synthetic

I have spent the last 25 years working in R&D at US Synthetic, a manufacturer of synthetic diamond tools for oil and gas drilling, machining, and bearing applications. I vividly recall reading the chess game analogy in Certain to Win by Chet Richards and wondering what we could do to improve speed in product development.

“Everyone wants to go ‘faster’ in development, but what does faster mean? Speed should refer to the rate at which we learn.” — Allen Ward

This focus on rapid learning was a powerful insight that shaped my journey with lean product and process development (LPPD).

Our product development process typically involves changing the material properties of the diamond to match the requirements of our customers’ applications. Almost every new product we develop must be fabricated and tested in our lab to verify that performance matches our design targets. Our designers were often waiting for parts to be built or lab tests to complete so they could learn if their ideas worked or if they needed to try something else. The time from concept to prototype to test result ranged from 30 to 45 days. I wondered what it would be like to wake up in the morning with an idea and have lab results before I went home that evening. The idea of one-day learning cycles became a simple, clear goal that we could use to drive improvement work. 

One of our first kaizen (improvement) events related to rapid learning focused on the prototype manufacturing step. At the time, highly skilled technicians would beg, borrow, and steal time on the production floor to build prototypes. The kaizen prep revealed that it took an average of 23 days to make prototype parts for testing. Simple tools like spaghetti diagrams and process flow maps showed that a dedicated prototype cell could dramatically reduce cycle time. 

We cobbled together an assortment of used and unwanted machines from the production floor and built our first cell dedicated to product development (Figure 1). We asked some of the technicians to staff the cell full time so we could experiment with the system and measure the improvement. By the end of the kaizen event, we had reduced the turnaround time from 23 days to four days and demonstrated the value of a full-time staff. Fortunately, our current prototype cell has the same state-of-the-art machinery as our production floor, and we have been able to further reduce flow time to 1.5 days through years of ongoing PDCA.

The next area we chose to tackle was testing. Our data showed that it took an average of 20 days to complete a test once the parts were built. The kaizen team value stream mapped the testing process and identified numerous sources of waste. Why did we walk nearly five miles back and forth from the testing bay to an office to input data into a computer for each test? Why did we shut the testing equipment down for 40 minutes each day to clean rock cuttings from a settling tank (Figure 2)? Why did it take nearly four hours to change over from one test to another? How were the specific test parameters chosen, and what would happen if we changed them? We tackled each of these issues through multiple PDCA cycles and eventually cut testing turnaround time from an average of 20 days to just under two. 

We have yet to achieve our goal of a one-day turnaround from idea to test, but we came remarkably close, reducing our learning cycle time from 45 days to around 4. That is nearly ten times the learning rate. Imagine the chess game if you could move 10 times for every move of your opponent! Think about what that means to a development team. They can now learn something new 65 times a year compared to six times a year with the previous system. 

Rapid Learning as a Competitive Advantage

Those of us involved in product development often face constraints like the ones in the chess example. Despite our best efforts, we are not always first to market with a new product or innovation. Our competitors may have more money, more people, and more resources working on product development than we do. Finally, the basic rule of all competition is to assume you are not smarter than your competitors (or your customers, for that matter). 

Is there a general strategy we can use to overcome these disadvantages and still win? Air Force Colonel John R. Boyd was obsessed with this question. Boyd was a fighter pilot and highly influential military strategist. The F-86 he flew during the Korean War achieved a stunning ten-to-one kill ratio against the MiG-15, a far superior aircraft in many ways. The MiG-15 could make harder turns than the F-86, could accelerate and climb faster, and had better high-altitude performance. So, what happened? 

Boyd identified two subtle advantages the F-86 had over the MiG. First, the F-86 had a bubble canopy that gave the pilot a 360-degree field of vision, while the MiG pilot’s view was blocked to the rear. Second, the F-86 had full hydraulic control, allowing the pilot to transition from one maneuver to another quicker than the MiG, which did not have hydraulic controls. Boyd hypothesized that the ability to learn and adapt faster in the F-86 was the secret to its success. 

Boyd’s interest went well beyond the F-86 paradox, however. His direct experience as a fighter pilot and considerable research ranging from the German Blitzkrieg of World War II to the Toyota Production System led Boyd to a breakthrough insight: we can use time as a principal strategic device. 

“The ability to operate at a faster tempo or rhythm than an adversary enables one to fold the adversary back inside himself so that he can neither appreciate nor keep up with what is going on. He will become disoriented and confused.” —John Boyd

Steven Spear makes the connection between learning rate and performance clear in The High Velocity Edge. The difference between good companies and great companies is their rate of learning. Toyota did not start out making high-quality cars at low cost. In 1965, Toyota’s productivity was about half that of GM (Figure 3). However, Toyota was learning how to improve quality and reduce cost at a rate about 3.5 times faster than GM so that by 1990, Toyota’s productivity was twice GM’s.

Toyota and other fast learning organizations have turned their rate of learning into a sustainable competitive advantage. If we can learn faster than our competition, we are certain to win!

“Competitive advantage derives from discovering new principles, or new applications of basic principles, specific to your products and obtainable only from your experience.” —Allen Ward

Unintended Consequences

Increasing our learning rate by a factor of 10 created several benefits for US Synthetic. First, we were able to test multiple parts instead of just one. Variation is a fact of life, and the only way to know the size of your measurement variation is to make repeat measurements. Being able to see the variation helped us avoid launching products that would not actually improve performance. Second, we could now use DOE (Design of Experiments) instead of One-Factor-At-A-Time experimentation to truly understand which factors improved performance. It became clear that we had multiple ways of achieving good performance. Some were more complex, costly, and of lower quality than others. The DOE results provided a rational framework for fixing the elements that did not affect performance while leaving the performance elements flexible. The quality of new products improved using this fixed-vs-flexible approach to reduce complexity and variety. Yields of new products have been greater than legacy products for the past four years!

I would like to say that is the end of the story, but the reality is a little more complex. One unintended consequence of reducing turnaround time by a factor of 10 was that it is now faster to test every new idea than to see if the data already exists. We have a gap in our ability to capture and reuse knowledge. However, I know we can close this gap with ongoing kaizen and by drawing on the LPPD community for ideas and inspiration.

An edited version of this post first appeared in the May 2021 Design Brief from the Lean Enterprise Institute.