Nature Works Through Interactions

Why the "Scientific Method" you learned in school is fundamentally flawed.

A recent study found that some common combinations of emulsifiers, dyes and artificial sweeteners were associated with a higher risk of Type 2 diabetes beyond what could be explained by the individual food additives alone. The International Council of Beverages Associations (ICBA) called the study’s results “misleading” and confusing for consumers:

Decades of robust scientific evidence show these ingredients are safe. Suggesting that recipes—a combination of safe ingredients—are worrisome is simply absurd.

Safe ingredients on their own does not imply safe ingredients in combination. The ICBA comment illustrates a lack of understanding regarding interactions that I believe is due, in part, to the way we teach science in American schools. In particular, the one-factor-at-a-time (OFAT) experimental methodology most students learn is blind to interactions. In biological and other complex systems, interactions are not a rare nuisance; they are often the most important part of the story.

What is an Interaction?

An interaction occurs when the effect of one factor is dependent on the level of another factor. Think about baking a cake. The ideal baking time depends on the oven temperature. Thirty minutes might be perfect at 350°F, but it will be too short at 300°F and will burn the cake at 400°F.

The natural world is full of interactions. Newton’s Second Law is a classic example in physics. The force required to move an object is the product of its mass and the desired acceleration. The effect of mass on the system is entirely dependent on the level of acceleration. If you are not accelerating, the mass of the object has no impact on the net force. However, as acceleration increases, the cost of that mass in terms of required force scales multiplicatively. Conversely, for a fixed amount of force, the resulting acceleration is inversely dependent on the mass. A heavier object resists the effect of the force more than a lighter one. Ohm’s Law and the Ideal Gas Law are additional classic examples of interactions in physics.

In chemistry, the Haber-Bosch process creates ammonia for fertilizer and is essential for feeding the world’s population. It relies on an iron catalyst interacting with nitrogen and hydrogen gases under high pressure and temperature. Without the interaction with iron, the reaction is too slow to be practical.

In medicine, a synergistic interaction occurs when the combined effect of two drugs is greater than the sum of their individual effects, like aspirin and caffeine for pain relief. An antagonistic interaction occurs when the interaction is harmful, like alcohol and antidepressants. Cancer patients are often treated with a “cocktail” of therapies, such as chemotherapy and radiation, or multiple chemotherapy agents at once because the combination treats cancer more effectively than any of the treatments alone.

Our genes don’t operate in a vacuum, either. Their expression is profoundly influenced by the environment, and this interplay determines traits and disease risk. In a famous study of adoptees, those with a high genetic risk for schizophrenia (their biological mothers had it) who were raised in a disturbed family environment had a 36.8% chance of developing the disorder. However, those with the same high genetic risk who were raised in a healthy environment had only a 5.8% chance, nearly the same as the low-risk group.

Ecological interactions between organisms and their environment are abundant. Bees get nectar from flowers while the flowers get pollinated, and fungi form mycorrhizal networks with tree roots, helping the tree absorb nutrients in exchange for sugars.

From the molecular level to entire ecosystems, nature appears to work primarily through interactions. So why did the ICBA claim interactions between food additives was absurd?

One-Factor-At-A-Time (OFAT)

Think back to your first, and perhaps only, exposure to the “scientific method.” Most likely you were taught something like the following:

1. Ask a Question

2. Form a Hypothesis

3. Conduct an Experiment

4. Analyze the Data

5. Draw a Conclusion

When it came time to conduct the experiment, you were taught to systematically hold every conceivable condition constant while altering just a single factor. Any subsequent change observed in the outcome could then be attributed to that lone, isolated change. This one-factor-at-a-time (OFAT) approach for controlling variables is deeply embedded in how science is defined and taught in American schools. Nearly all science fair guidelines, including those from NASA, explicitly state the requirement to test only one factor at a time.

While not completely wrong, OFAT experimentation suffers from serious weaknesses: it completely ignores factor interactions, it isn’t efficient or thorough, and it is highly subjective. The problems with the OFAT method were identified over 100 years ago by Sir Ronald Fisher.

Fisher’s Insight

In 1919, Ronald Fisher was hired by the Rothamsted Experimental Station in Hertfordshire, England, to make sense of the vast amount of data from their Classical Field Experiments, some of which had been running continuously since 1843. Fisher recognized that the nature of agricultural research imposed constraints that made OFAT experimental methods deeply problematic. The primary limitation was time. An experiment testing the effect of a new fertilizer on crop yield took an entire growing season to complete. Testing several different fertilizers and crop varieties one by one would take many years, a timescale that was both scientifically and economically untenable.

Compounding the issue of time was the problem of uncontrollable variation. An agricultural field is a heterogeneous system. Slight differences in soil, drainage, sun exposure, and pest infestation could introduce variability or “noise” into the results. This background noise could easily be larger than the actual effect of the treatment being studied, making it impossible to determine if any improvement was due to the treatment or simply due to that plot being on a slightly better patch of soil.

The need to get reliable answers from a limited number of trials in a noisy, complex environment created the imperative for a new science of experimental design. As Fisher delved into the Rothamsted data and began designing new experiments, he came to a conclusion that placed him in direct opposition to the established scientific wisdom regarding OFAT. He articulated this conclusion in his 1926 paper published in the Journal of the Ministry of Agriculture:

No aphorism is more frequently repeated in connection with field trials, than that we must ask Nature few questions, or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken. Nature, he suggests, will best respond to a logical and carefully thought out questionnaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed.

Fisher’s critique was not a minor quibble about technique. He was proposing a comprehensive dismantling of the logic that underpinned the traditional OFAT approach. His argument rested on three interconnected flaws he identified in the OFAT methodology.

1. OFAT Cannot Identify Interactions

Fisher’s most profound critique was OFAT’s fundamental inability to detect or measure interactions. OFAT varies only one factor at a time and assumes that interactions do not exist. This is a fundamental structural limitation. There is no systematic way to find an interaction with OFAT because the data required simply doesn’t exist.

Going back to the cake baking example, the problem is to produce a good tasting cake. The chef decides to experiment with oven temperature using the OFAT approach, holding the baking time fixed at 30 minutes along with all other known factors (type of oven, type of baking dish, altitude, etc.). The cake was raw at 300°F and burnt at 400°F. The chef decides the best temperature to bake a cake is 350°F. The effects of the other factors are not measured. There is no data showing how temperature and time interact, so there is no way to see that a good cake could be obtained by extending the baking time to 60 minutes at 300°F. Fisher realized that in a world governed by complex interdependencies, a method that cannot see them is not just limited, it is a recipe for error.

2. OFAT is Inefficient

The most practical flaw Fisher identified was OFAT’s inefficiency. In an OFAT experiment, each trial provides information about only one factor. Fisher argued this was a waste of resources in the context of agricultural trials where each run took a full year. His proposed alternative, what became known as the Design of Experiments or DOE, allows a researcher to investigate multiple factors simultaneously.

The key insight is that in a DOE, every single trial provides information on every factor being studied. For example, in a simple experiment with two fertilizers and two crop varieties, a factorial design would have four treatment groups:

To estimate the main effect of the fertilizer, one compares the average yield of the two Fertilizer 1 experiments (group 1 and 2) with the average yield of the two Fertilizer 2 experiments (group 3 and 4). Crucially, the entire experiment (all four groupings) contribute to this estimate. Similarly, all four groups contribute to the estimate of the effect of crop variety. Fisher argued one could learn as much about many factors as one could learn about a single factor in the same number of trials, dramatically increasing the efficiency and power of the experiment.

3. OFAT is Subjective

Because the OFAT method explores the potential experimental space one dimension at a time, it can easily lead the researcher to a “false optimum.” The result of an OFAT optimization process is entirely dependent on the initial starting conditions and the order in which the factors are tested. In other words, the results are subjective and rely on luck.

Think about climbing a mountain in thick fog. The climber starts at base camp and walks in one direction (say North) until the altitude no longer increases. This is the local North-South peak. From that point, they walk in another direction while holding the first direction constant (say East) until the altitude no longer increases. The climber now believes they have reached the summit. However, the true summit of the mountain might lie to the Northeast, a direction they were forbidden from traveling. By only moving in cardinal directions (one variable at a time), they have trapped themselves on a lower ridge, a false optimum. Had the climber started at a different base camp or chosen to explore East-West first before North-South, they might have ended up on a different ridge entirely, potentially closer to or further from the true peak.

Fisher found this path-dependency “very unsatisfactory.” A reliable scientific method cannot produce different optimal solutions based on the arbitrary order of operations. It must be able to survey the entire landscape to find the true peak. The OFAT method, by its very nature, explores only a narrow, cross-shaped path through the experimental space. The unthorough investigation leaves vast areas in between, where the true optimum often lies, completely unexplored.

Consequences of OFAT Thinking

The ICBA’s mistake is an unavoidable consequence of OFAT thinking. An experimental system that is structurally unable to detect interactions, too inefficient to find them even if they exist, and entirely reliant on arbitrary starting conditions will naturally dismiss any claims of complexity. This is precisely why the combined effects of food additives were declared “absurd.”

Fortunately, there is a better way. Fischer’s work spurred him to develop new methods of designing experiments that could simultaneously provide answers to many questions, most of which could not even have been asked using one question at a time. Fischer’s methods are commonly referred to as the Design of Experiments or DOE. DOE transformed statistics from a descriptive tool for summarizing data into a prescriptive and inferential science for generating new knowledge. The experimental design is of paramount importance because no amount of information can exceed the quantity supplied by the data. If the data doesn’t exist, the effect cannot be measured.

The Design of Experiments (DOE) framework enables you to move past subjective, inefficient, and blind experimentation, accelerating your innovation. If you would like to learn more about how to incorporate DOE methods into your experiments, please reach out to me at ken@objexp.com or see objectiveexperiments.com.