When we do research, we can make many errors. Some of them occur often enough to have names. Here are nine common ones
Here they are again, in more detail, with some ways to try to avoid them:
1. Selective observation
Selective observation happens when our attention is drawn to answers or observations that confirm our pre-existing beliefs. It's a lot like selective hearing (i.e., when people, especially children, hear only the things they want to hear). For example, if I hypothesize that blacks are more likely than whites to speed, I am probably likely to note the blacks who are speeding while paying less attention to speeding whites and blacks who are not speeding. A better approach to this study would be to write down the speed of every car going by and the race of the driver. I could then make tables and compare the percentages of speeding drivers of each race. Chances are that I'll find that race isn't related to one's likelihood of speeding.
A few ways you can try to avoid selective observation in your research are to do a literature review (so you'll know which relationships other researchers found), decide your research approach beforehand (e.g., when I decided above to write down the speed of every car rather than just depend on my memory), take thorough notes (to prevent biases from affecting your memory), watch for "disconfirmatory" information (such as speeding whites and non-speeding blacks), and consider both "sides" of your study (i.e., try to argue against your hypothesis; if you can't poke holes in your theory then you're more likely to be on the mark).
You could also use time or area sampling. Time or area sampling means that you focus your attention on a smaller part of the action for a given amount of time. Instead of trying to watch the entire crowd at a basketball game, for example, I could look at the rightmost four columns of people for ten minutes, then the next four columns for ten minutes, and so on. By doing this, I would be able to get a picture of what the whole crowd was doing. And, most importantly for selective observation, I would be forced to look at all parts of the crowd at some point time, rather than just those who were doing what I expected them to do.
2. Inaccurate observation
Inaccurate observation happens when we "misremember" or misrecord data. How many times have you missed a question on an exam because you incorrectly copied down something from lecture? That's one form of inaccurate observation. You thought you correctly observed the information when you really hadn't done so. Have you ever misunderstood what someone said, and thought s/he said something that rhymed with his/her actual utterances? That's another form of inaccurate observation. Your brain somehow miscoded the information at the processing stage. Inability to remember what you saw is another form. Your brain somehow jumbles or changes the original memory during the recall stage.
If you plan to observe (and possibly even take notes), you can increase your accuracy. Planning includes focusing on the task at hand rather than daydreaming during data collection; this will help prevent many errors. Doing a literature review can help by alerting you to what you might find (that way you won't be as surprised and will be better able to process and recall the information you see or hear). Other techniques include using forms (e.g., where you circle people's characteristics rather than trying to write them down; forms also "guide" your observations to include all the important information points), doing time/area sampling (to reduce the amount and variety of information you are responsible for writing down and remembering at any given moment), and writing down as much information as you can (you should assume that anything you don't write down will be forgotten or remembered incorrectly). Another suggestion is to practice observing and recording your observations before actually doing it for real. Practice will make it easier to quickly and accurately record or summarize a given situation, and will show you the parts of the task about which you are less clear.
3. Overgeneralization
Overgeneralization is generalizing to others who are different from one's research population. This happens all the time. A study in New York City finds a high proportion of gang members and school administrators in rural America panic and institute draconian measures intended to stem the proliferation of gangs in their districts. A parenting program works in one community, so planners automatically assume it will work in theirs. A new study program raises grades for high school students, so colleges clamor to include it in their curriculum. All three of these examples illustrate how quickly some people take research findings as absolute. Who knows, maybe the findings actually do generalize. The problem is that you can't make that assumption.
Just because a program works in one community doesn't mean that it will work in another. For example, consider the NCI Alcohol Treatment program in Navajo country. Alcohol abuse programs "borrowed" from dominant society didn't have very high success rates for American Indians, but that didn't stop social service agencies from using them anyway. Then, NCI incorporated traditional practices and ideas (e.g., use of certain ceremonies) into their program which works much better for American Indians. Not ironically, non-Indians (and many Christianized American Indians) don't find the program to be very useful. This example illustrates overgeneralization because the social service agencies assumed that dominant society treatment programs would work equally well for American Indian clients. After all, they worked for whites! The social service agencies were overgeneralizing; that is, they were assuming that what works for whites will work for others. In the end, they learned that what's good for the goose is not always good for the gander. There might be important differences between populations that will affect the success of a given program.
Some ways to avoid overgeneralizing:
4. Made-up information
Made-up information happens when one fills in details without a scientific basis for doing so. As researchers, we have to fill in a lot details; this is called inferring. The problem is that some inferring isn't based in science; instead, it's based in stereotype and speculation.
In one of my research classes long ago, a classmate observed and reported to us the following: an attractive young woman was sitting alone in a bar for a few minutes when a man walked in. The two talked briefly, then left together smiling. What happened in that situation? At first, my classmate thought it was a typical "pick up" and may have even been the solicitation of a prostitute (especially since neither consumed any beverages). When he saw this happen day after day at the same time, however, he approached the young woman and asked her to explain what he had been observing. It turned out that she worked in the area, and merely came to sit in the bar to wait for her brother to pick her up after work. Had my classmate just made up the information, he would have assumed a very different situation!
One of my students observed a number of student government meetings and concluded at first that the council was unprofessional and didn't respect each other's views. She based this conclusion on the fact that throughout all the meetings, they were passing notes, especially during discussion of what she felt were important issues. It later turned out that the students were merely following Robert's Rules of Order, and were writing their names on slips of paper to be passed to the student who maintained the speaker's list. He would add the names from the slips of paper onto the bottom of his list and call on them in turn. Oops, major misunderstanding. Had she asked earlier, she would have saved herself a lot of embarrassment! Made-up information may or may not be correct.
For wonderfully humorous examples of this problem, consider Horace Miner's contention that the Nacirema are a "magic-ridden people" who are destined to self-destruct due to their mystical beliefs (take the time to enjoy the article at: http://www.msu.edu/~jdowell/miner.html), and an unknown scholar's equally clever discussion of the Asu's "overwhelming preoccupation with the care and feeding" of their sacred racs (read the article at: http://everything2.com/index.pl?node_id=999030). In case you haven't figured it out, the Nacirema and Asu are 'Americans' (both are simply spelled backwards to baffle the reader) and the writers are poking fun at anthropologists who misinterpret what they see. By far the best example of made-up information is to be found in David Macaulay's Motel of the Mysteries (1979, Boston: Houghton Mifflin Company), a futuristic look at an excavation of the Toot'n'C'mon Motel in the year 4022. Here's a sample of the misunderstandings that the future archaeologists made, this text describing a common credit card found in one of the rooms:
This extremely fine piece of workmanship served as a portable shrine which was to be carried through life and into eternal life. Its delicate inscriptions were intended to identify an individual's religious preference along with the burial site to which the body should be delivered when necessary. Matching inscriptions were found on the main doors of the sanctuary. Because the ancients were unable to predict the exact time of death, each of the shrines had to last for an entire year (p. 64).Instead of making up information, you could do a variety of things. First, you could do like my classmate above and ask someone who knows what's going on for their opinion. If you're going to be in a strange or new setting, you could do a thorough literature review to help you understand possible outcomes and unique customs (this would have helped my student understand the note passing behavior she observed). Most of all, you should rely on prior research studies and/or theory to guide your own interpretation of what was going on.
5. Ex post facto hypothesizing
Ex post facto hypothesizing happens when a researcher decides what happened after it happened and after the study was done. In scientific research, we have to decide what will happen before we do our study, not after. What this means is that our research must be guided by theory, and that we have made predictions (hypotheses) about what we will find. It's similar in some respects to Ex Post Facto laws (laws that have been enacted after the offensive act took place), and are just as repulsive, at least to researchers.
This doesn't mean that one cannot use data that has already been collected by others. Assume, for example, that you wish to research whether election outcomes are based on public perceptions about the economy. You could obtain from a data clearinghouse, election data from the past. Then, you could somehow assess public perceptions of the economy at the time of each election (through polls from the era, editorials in newspapers, or some other method). You could then correlate the two and test your theory. What differentiates your study from a common case of Ex Post Facto Hypothesizing is that you made your predictions, then tested them. You didn't form your hypothesis before-hand.
The ways to avoid this problem in your own research are simple. First, remember that correlation does NOT imply causation; in other words just because the birth rate is high in areas with lots of storks doesn't mean that the storks are bringing the babies. Then, don't feel obligated to report your research as deductive (that is based on hypothesis testing). If you were simply exploring a topic, own up to it. Exploratory research is very helpful in establishing foundations for future scholars.
6. Illogical reasoning
Illogical reasoning is just that: illogical. It ranges from a bit off the mark to absolutely absurd. For example, we have all heard that bread always lands buttered-side down, but seldom question where that knowledge came from. I've actually heard of someone who proposed (rather humorously) that if one took a piece of buttered bread and tied it to the back of a cat and dropped the resulting mass, that the two diametrically opposing forces would cause the bread and cat combo to ceaselessly spin several inches from the floor. It's illogical to assume that the butter side will always land down. I've actually done research that shows that bread (and bagels and crackers and other such things) will land buttered-side up as well (now you know what to do with that horrible dorm toast). Now, the cat is another story; research has repeatedly shown that cats somehow miraculously upright themselves from even rather short falls (just take my word for this; don't try it on the family feline).
We can also make logically flawed assumptions in research. Medical researchers in 1879 argued that masturbation causes insanity and recommended exercise to the point of fatigue as a treatment (Medical and Surgical Reporter, Vol. XLI, 1879, p.542). The same journal contains research that claims that insanity can be contagious if people live together (p.13). In other words, simply masturbating or marrying someone with a history of insanity could make you insane. Despite its lack of logic, this research does add new meaning to Rosenhan's research (1973, Being Sane in Insane Places) that showed that insanity was assumed for anyone who had been admitted to the 12 mental hospitals under study. As a side note, the journal also includes a writeup on coffee as a cure for typhoid fever (p.39). Pretty illogical, eh?
P.S. Just because you reason logically doesn't mean that you're right about what you saw. Think about all the times you've made wonderfully sound assumptions about outcomes and been wrong. We can't all be like Sherlock Holmes and be right every time.
Two ways to avoid illogical reasoning in your own research are to base your decisions on prior research and theory, and to make extensive use of peer review. Peer review is when you have others read and critique your writings; this helps scholars find places where they've reasoned illogically.
7. Ego involvement in understanding
Ego involvement in understanding happens when people let the human side of them dictate their findings and how they view findings by other researchers. Those who are members of a researched group (e.g., sorority, family, or religion), for example, may find it difficult to explain what happens without bias; they would feel obligated to protect the "deep, dark secrets" of the group. They may feel compelled to justify even sinister behavior.
Ego involvement can also affect research on a person's "pet project." If I've just spent ten years developing a new treatment program, someone else should do the evaluation because I might be tempted to fudge results here and there, or at least misinterpret things in a way that is favorable to my program. After all, I worked hard on the program; it must really work! Those whose jobs depend on the program should also be excluded from the evaluation team because it would be hard to disband even an ineffective program that provided one's salary.
Sometimes ego involvement affects how we look at and react to research by others. Studies whose findings oppose what we believe are easily dismissed despite their rigor, whereas even shoddy studies that support our viewpoints are cited ad nauseam. Unfortunately, ego involvement by journal editors sometimes determines which studies get into print.
The best way to avoid ego involvement is to try to remain neutral, or to stay away from topics about which you can't be neutral. Be honest with yourself and refrain from research on topics that might be too close to home for you. Another suggestion is to use the team approach. It's harder to get too ego involved when sharing the research task with others who don't have your intense views on a given topic.
8. Premature closure of inquiry
Premature closure of inquiry occurs when we decide that we know enough about a topic and decide that it no longer warrants future study. Some topics are fairly easily dismissed as trite (e.g., science is now certain the earth revolves around the sun). Other topics, however, are still misunderstood (e.g., the causes of crime and insanity). For these less understood topics, it's important to keep researching until we more fully understand them.
Sometimes research is too controversial to get funding or support, regardless of its importance. Studies that parallel those by Stanley Milgram (who did the shocking experiments on obedience to authority) and Philip Zimbardo (whose Stanford prison experiment had to be called off early due to psychological trauma experienced by some of the participants) may have difficulty getting support in today's world. Likewise, medical research like that conducted by notorious Nazi Josef Mengele (whose fascination with flawless research methods led him to literally murder set after set of twins in the name of research) has been condemned by all civilized nations.
The best way to deal with premature closure is to keep looking for answers, even if it involves using different approaches. While the findings of Milgram's and Zimbardo's studies were valuable to society, the toll taken on their subjects was too great. Mengele's methods were simply satanic at best. Contemporary researchers shouldn't quit doing research on medical issues, obedience to authority, and socialization of prison inmates and guards; instead, they should find non-controversial ways to do it. This may require some serious thought and planning. Some contemporary researchers, for example, have studied obedience to authority by having people endlessly tear paper into little squares. Their willingness to comply with this task causes far less stress than the belief that they were administering sometimes fatal shocks to another human being while still allowing us to study the fascinating phenomenon.
9. Mystification
Mystification happens when we attribute results to the supernatural. In olden days (and sometimes to this day), illnesses were blamed on a variety of spirits and deities. Trying to find a cure for cancer might have lead early researchers to see their local clergy! Crime, attributed to the devil's influence, was felt to result from lack of religious training.
Sometimes, the people we study will use mystification. One of my students, for example, asked ranchers about the mutilation of their livestock. To his surprise, a number of them insisted that the damage was caused by space aliens. Rather than report this as the actual cause, he kept looking for more sensible answers.
The ways to avoid mystification are similar to those mentioned above for other errors. First, keep looking for answers when you feel tempted to rely on enchanted explanations. Second, peer review will prevent you from embarrassing yourself when you insist that witches have afflicted young girls in Salem, Massachusetts.
Recap Q1: I found a juicy dataset and accidentally found that serial killers all had telephone numbers with the number "13" in them. I then decided that fear of the number "13" makes people into serial killers, wrote a book about my finding and became famous. What error of human inquiry is NOT present in this scenario?
A. ex post facto hypothesizing
You have now completed the section on "Common errors made in research." Click here to return to the main menu.
Page last updated: April 15, 2008