On-Farm Research: General Concepts - Summer 2002
Special Supplement on On-Farm Research

By Sue Ellen Johnson & Tom Morris
.pdf version (40 KB)

This is an overview of useful concepts for designing and executing a credible research project. Working with a Farmer Research Group and a scientist will help assure that your efforts and results are meaningful to you and to others. You should ask yourself who will be interested in your results; and what level of understanding and certainty do you want from your research or experiment? These considerations will guide you in designing your research. Designing an effective study takes time! Allow several months to think and plan!

Research Topics and Treatments

The first step is for the farmers to decide on a general topic or question for investigation. The next step is to identify the specific questions you wish to answer or practices you want to compare, test or evaluate. The question should guide you in selecting one or two particular treatments that will help you to answer your question. More than two test treatments can be difficult to manage. Select treatment(s) you can implement with precision.

Three Key Concepts for Valid Research: Replication, Randomization and Documentation

Three key concepts of field research are replication, randomization and documentation. You can be confident that your results will be both credible and valid to other farmers and to scientists if you understand and use these concepts. These concepts allow you to draw accurate conclusions from your research. Your results can be presented to any audience with confidence.

Replication is the repetition of your comparison or test in space and time. Repeating your experiment and treatments at several places and times allows you to be sure your results are due to your treatments, not just chance. Production naturally varies between individual plants, beds, animals; a replicated study lets you determine if the differences you measure are due to your treatment or just chance natural variation. The effect of a treatment on animals or plants has to be greater than the natural variation for you to know that your treatment made a real difference. Replication is needed to determine if differences between the treatments you are evaluating are large enough to be meaningful. A minimum of three replications is usually recommended. Replication on each farm assures valid conclusions for that farm. Whenever possible, replicate side-by-side within fields, and work with other farmers to have the same experiment and treatments across several farms. Repeat the experiment for more than one year. This makes your results and conclusions more valuable because they've been tested under a wide variety of conditions and managements.

Randomization is used to avoid any bias that might influence or skew the outcome of your inquiry. Bias can occur because of natural factors. Bias can also be introduced by the farmer or scientist. Assigning and implementing treatments randomly to replicates in a field or animals in the herd is the only objective way to make a fair comparison or evaluation.

Documentation is recording how you implemented your experiment and what happened during the experiment. In a research project, you need to record your measurements and observations, in addition to "just seeing what happens" (which may be interesting, but is not conclusive!). Preparing a map or plan for your research will reduce errors and increase the likelihood of a successful experiment. Preparing a record sheet will help you prioritize what you really want to learn and measure. Record your field actions, your measurements and observations as you make them. There are no uncorrectable errors or mistaken interpretations if there is an accurate record of what happened in the field during the experiment.

Replication and randomization are the prerequisites of any systematic inquiry. Documenting your research- both the implementation of your experiment, and your results is essential. If you don't record your measurements, they don't exist. Without replication, randomization, and documentation making measurements isn't worthwhile.

More concepts that will affect your research results and their interpretation

Measurements versus Observations
As you plan your experiment, you need to decide what you need to measure. The actual measurements and observations you make are determined by your topic and the questions you are trying to answer. You will want to measure -physically and numerically quantify —certain responses or qualities; other responses you may only observe (for instance color of the fruit, soil drainage, insect damage, feed palatability). Measurements are different from observations. Measurements add certainty and validity to your observations; so plan measurements carefully. A measurement is the weight of tomatoes from a plot; an observation is that there are a lot of weeds in the mulched treatments. Both measurements and observations provide information that help you interpret the outcome of your study. Observations give you cues about what you need to measure in the future. Both measurements and observations take time. Of course, everything of interest in your study may not be measurable. Although it is not practical or possible to measure everything, measurements are critical in order to draw conclusions from your research. Some things can't be measured cost-effectively. Avoid taking measurements you won't use, but remember that observations only provide clues, while measurements lead to conclusions.

You can choose to make measurements and observations only at the start and finish of an experiment or you can make them periodically during the experiment. Some of the observations you make will be incidental, but you should have a systematic plan for making both measurements and observations. Planned, systematic measurements and observations will keep your research interesting, help you manage your time commitment and make it easier to follow-through. Example Your thoughtful, systematic measurements and observations — whether they are biological or economic — will be of keen interest to other farmers and scientists.

Baseline Measurements
Baseline measurements are taken at the outset of an experiment, before you implement your different treatments. Baseline measurements enable you to track changes through time. Good baselines define the status of each of your replicates at the start of the experiment. Baseline measurements taken from every replicate indicate whether conditions were the same or different across the study sites when you began the experiment. A lot of variation among the baseline measurements taken from each of your replicates indicates it may be difficult to prove changes were caused by a treatment. On the other hand, any differences that show up consistently across highly variable replicates are usually important differences. Baseline measurements may not be required in your study, but they can be useful.

Controls
A control treatment is critical for most studies. Control treatments are replicated side-by-side in the field or barn with the new practice. The control treatment often is your typical or standard practice. The control helps you to interpret your results by comparing the performance of a new practice relative to your usual practice. For example, you might compare your own standard compost with another compost including poultry manure. Sometimes the control is where no treatment is applied whatsoever. For example, you might compare your own compost at your usual rate of application with no compost at all, or at twice the rate you usually apply. You may not need a control treatment in your research, but most on-farm experiments are improved by a control treatment.

Confounding
Unless you plan and execute your research carefully, you face the prospect of confounding your results. Confounding is when one factor interacts with another factor and influences the interpretation of the results. (You may think your treatment made a difference when the difference was actually caused by another factor entirely). Confounding can be caused by trying to test too many factors at once or poorly delineated treatments, which is why it is usually more straightforward to only test a specific practice or idea in a particular experiment.

Consistency and Variability
It's important to be consistent as you implement your study. The entire experimental area or all the animals in an experiment need to be managed the same way. The only thing that should be implemented differently within your study area is your treatments. And, you must implement each of your treatments the same way in each of the replicates. A lack of consistency, or too much variability, can obscure differences between treatments. You may think a treatment makes no difference when it does. The more variable the conditions within your experimental site, the more difficult it is to detect small differences between your treatments. Sometimes small differences can be economically and environmentally significant for your farm. Obviously, you can't standardize everything in an on-farm research project, but you want to avoid adding to the farm's natural variation. Managing "experimental" variability in an on-farm system may seem difficult, but it is possible. Managing variability is where a Farmer Research Group and a scientific, statistical approach can be helpful.

Selecting an on-farm research site
The area (or animals) you use for your study depends on what you're investigating. Choose a study site that represents your farm's environment, soils and situation. The site should not have too much variability. Use an area (or number of animals) big enough that you will believe your results, but not so big that you are taking a great risk if the experiment fails. You need room to replicate and randomize your treatments at your study site, in the same field. It helps if the site is accessible and visible.

Remember
The goal in designing a research project is to define questions and select practices for systematic, organized inquiry. A poorly defined question or poorly designed study makes it difficult to obtain understandable results that answer your question. (the "maybe it was just the weather" phenomenon). You want your research design to account for the complexities of your farm ecosystem, and the farm's logistical and labor constraints. Work with other farmers and a scientist to design an effective on-farm research project that is meaningful to you, farmers and scientists.

Back to Summer 2002 TNF Page

The TNF is the quarterly publication of the Northeast Organic Farming Association. Click here to learn more about the TNF.

This page was last modified on March 06, 2004 at 9:07:35 PM.

Home | About Us | State Chapters | NOFA Interstate Calendar | NOFA Summer Conference | Search the Site
NOFA Video Project | Farmer to Farmer Exchange | NOFA Interstate Council | The Natural Farmer | Online Store