Designing the Scientific Study

Gaining scientific knowledge about the natural world requires a researcher to carefully follow a series of steps that ensure the rigor and reliability of the results (Figure 1.6). This section will detail each of these steps.

The scientific process typically starts with an observation (often a problem to be solved) that leads to a research question that the study is attempting to answer. Let’s think about a simple problem that starts with an observation and apply the scientific method to solve the problem. One Monday morning, a student arrives at class and quickly discovers that the classroom is too warm. That is an observation that also describes a problem: the classroom is too warm. The student then asks a question: “Why is the classroom so warm?” Recall that a hypothesis is a suggested explanation that can be tested. To solve a problem, several hypotheses may be proposed. For example, one hypothesis might be, “The classroom is warm because no one turned on the air conditioning.” But there could be other responses to the question, and therefore other hypotheses may be proposed. A second hypothesis might be, “The classroom is warm because there is a power failure, and so the air conditioning doesn’t work.” A hypothesis must be testable to ensure that it is valid. For example, a hypothesis that depends on what a bear thinks is not testable, because it can never be known what a bear thinks. It should also be falsifiable, meaning that it can be disproven by experimental results. An example of an unfalsifiable hypothesis is “Botticelli’s Birth of Venus is beautiful.” There is no experiment that might show this statement to be false.

Interpreting the Results

The graph summarizing the data (Figure 1.6) shows a large difference in algae growth between treated ponds (less growth) and control ponds (more growth), which supports her hypothesis. When a scientist finds support for her hypothesis, she does not say that this explanation is “true” because she knows that future experiments could show a different result, perhaps because of a variable that did not occur to her. For instance, the someone else may discover that the phosphate reacted with the lining of the artificial ponds, releasing a different toxic chemical that suppressed algae growth. If, instead, the data did not show a difference in algae growth, then she would reject her hypothesis. Be aware that rejecting one hypothesis does not determine whether or not another hypotheses can is supported (e.g. rejecting the the hypothesis that phosphate suppress algae growth does not mean there is support for the hypothesis that calcium carbonate suppresses algae growth). It simply eliminates one hypothesis or explanation.

Other Important Aspects

There are a few other critical aspects of scientific research that must be considered to ensure that a scientist can be confident that a hypothesis is either supported or rejected.

First, there is the issue of how research subjects are chosen and placed into either the experimental group or the control group. Research subjects are the units of interest in the study – they could be ponds that could grow algae, plants that could grow taller with a new fertilizer, or humans whose health could be improved with a new medication. Let’s say that a researcher is studying the effect of a new medication that is hypothesized to reduce blood pressure. He has access to several research subjects – people who have high blood pressure. He must determine which individuals will receive the new medication (experimental group) and which individuals will not receive medication (control group). Among his pool of research subjects, he notes that about half the individuals are female and about half of them are male. He also notes that they range in age – about half of the subjects are 40-55 years old and the other half are 56-70 years old. When determining which subjects should be in which group, he must be careful not to place all of the males in one group (for instance, the experimental group) and all of the females in the other group (for instance, the control group). Why? Because he will not be sure if the effect that he potentially finds is due to the new medication or if it is due to the differences between females and males. The same issue applies with age: if all of the younger individuals are in the experimental group and and all of the older individuals are in the control group, the resulting effect could be due to age rather than the medication. Thus, research subjects must be divided randomly according to their characteristics (such as sex and age). We call this process randominization – randomly assigning research subjects into either the experimental or control group. Imagine a deck of playing cards in which half the cards are red and half are black, and they also vary by number. To randomly sort the cards so that all of the red cards do not end up in one pile or all the low cards do not end up in one pile, you could shuffle the cards several times and then sort them (deal the cards) into two piles (Figure 1.8). Then, the two groups will be much more likely to have sets of similar characteristics – similar numbers of males and females, similar age range. The research will be much more likely to focus on the variables of interest to the study – i.e. the independent and dependent variables – rather than the other extraneous variables like age and sex.

Second, there is the issue, in the case of research on human subjects, of researcher and subject expectation in which the participants in the study can create a bias that could skew the results. A human subject is aware that a research study is taking place and has consented to participate. This knowledge can create a psychological bias in which the subject expects some effect and thus imagines or creates this effect. Just imagining that one is taking a medication that could lower blood pressure could actually lower blood pressure because it might, for instance, decrease the psychological anxiety over having high blood pressure. It is impossible to completely mitigate this bias. However, there are ways to minimize it. One of the easiest ways to reduce bias is to not inform the research subjects whether they are in the experimental or control group. In this case, the subjects in the control group must be given a placebo, which is “fake” treatment such as sugar pills that have no effect on their health. The subject must not know whether they are receiving, for instance, the experimental medication or a placebo. This psychological bias can also be present in the researcher. The researcher might unconsciously treat a subject differently depending on whether the subject is in the experimental or control group. In many large clinical trials of a new medication, the head researcher never meets the research subjects. Instead, research technicians meet with and examine the research subjects, and these technicians are also unaware of which subjects are receiving treatment and which subjects are receiving a placebo. This aspect of research design, in which both the research technician and the research subject do not know which group they have been placed in, is called double-blind (Figure 1.9).

Third, analyzing data often requires more than just visualization on a graph. The data could show a difference simply due to chance. Statistical tests are one way to reduce the chances that the results are simply due to chance rather than the effect of the independent variable. Statistics in a discipline that involves the organization, analysis, interpretation of data. The calculation of “average amount of algae growth” in the example above (Table 1.1, Figure 1.6) is an example of a statistical operation. More sophisticated statistical tests allow a researcher to determine, with a high level of confidence, that a difference between two averages is due to some factor (such as the independent variable) rather than by chance. Imagine a researcher is conducting a study to determine whether a fertilizer has an effect on growth of a particular plant species. The independent variable is the amount of fertilizer and the dependent variable is the amount of plant growth. Notice that once the data are collected and the average amount of plant growth is calculated, the ranges of plant growth in experimental and control groups overlap and the averages for each group are very close (Table 1.2, Figure 1.10).

Table 1.2.  Data on plant height growth in plants after 4 weeks
of treatment with a new fertilizer (treatment group) and
plants that have were not given any fertilizer (control group).

How does the researcher know whether this slight difference in averages is not simply due to chance? There are statistical equations that can be applied to the data that will inform the researcher of the likelihood that the difference is due to some effect rather than chance. These statistical analyses are never absolute – they can never tell the researcher with 100% confidence that the difference is not due to chance. But, they can provide a very high likelihood that the results are due to some real effect, which is called statistical significance. In the plant growth example, the statistical analysis shows with 99% confidence that the difference in averages between the experimental and control groups is not due to chance. Thus, these results provide support for the hypothesis (or, they could provide a reason to reject a given hypothesis). In sum, analysis of scientific data includes a summary of the data (calculating an average is an example of a summary statistic), a presentation of the results (in tables and graphs, for instance), and various statistical tests for significance differences.

Fourth, another important aspect of research design is sample size. The larger the sample size, the less likely that the results are due merely to chance. For example, a researcher hypothesizes that the students in first grade at School A are taller on average than the first-grade students at School B because School A is in a wealthier neighborhood so they have access to better nutrition. However, it would take far too much time and effort for the researcher to measure all 128 first-graders at School A and all 153 first-graders at School B. She must take a sample of students from each school. If she only measures two students per school, she could easily by chance measure two particularly short first-graders at School A and two particularly tall first-graders at School B, or vice versa. She must have a larger sample size to be more confident of her results. Many statistical tests have sample size built into the equation. Thus, in order to get a result that a researcher can be very confident is accurate (or statistically significant), then the researcher must have a large enough sample size.^[1]

Fifth, there is a crucial issue that must be considered when analyzing and interpreting the results of a study. The two variables of interest – the independent and dependent variables – may be correlated but that does not mean that they are causally related. Correlation means that two variables are related – they vary at the same time and in a consistent manner. For example, in the case of the algae pond experiment, the presence of phosphate is correlated with a lack of growth in algae. In the case of the plant experiment, the application of the fertilizer is correlated taller average height of the plants. Causation means the one variable (presumably the independent variable that is controlled by the researcher, such as presence of phosphate or fertilizer) causes an effect in another variable (presumably the dependent variable). However, the correlation could be due simply to chance – a coincidence. Also, the correlation could be caused by a third (perhaps unknown) variable. For example, the rise in average global temperature is negatively correlated with the number of pirates in the world (Figure 1.11). Does this mean that the increase in global temperatures is caused by a lack of pirates? Or, that pirates have been disappearing because of the rising temperatures? Not likely. The rise in global temperatures is more likely caused by increased carbon in our atmosphere, whereas the decrease in the number of pirates is due to stronger ship construction and better security. Arguably, both of those causes are ultimately caused by increased global industrialization over the past 100 to 150 years.

In sum, though no correlation certainly indicates no causation, the presence of a correlation does not always indicate causation. Determining whether a correlation is an indication of causation can require a study that includes multiple variables instead of just two variables. In addition, scientific studies should be repeatable by other researchers and this repetition can help sort out correlation versus causation in some cases. For instance, another researcher might test a similar hypothesis with some changes in research design (e.g. another sample of research subjects with different characteristics) and find similar support for the hypothesis.