psy 496 for samantha only

psy 496 for samantha only
Applied Project Capstone in Psychology R. Eric Landrum Boise State University Bridgepoint Education, Inc. lan66845_00_fm_i-xiv.indd 1 4/20/12 2:36 PM R. Eric Landrum Applied Project: Capstone in Psychology Bridgepoint Education, Inc. VP of Learning Resources: Beth Aguiar AVP, Editor-in-Chief: Erik Evans Editorial Director: Steve Wainwright Director of Editorial Technology: Peter Galuardi Sponsoring Editor: Mireille Yanow Development Editor: Carrie Brandt Assistant Editor: Kristle Maglunob Editorial Assistant: Nicole Sanchez-Sullivan Media Editor: Kim Purcell Printing Services: Bordeaux Composition/Illustration: Lachina Publishing Services Cover Image: Diana Ong/SuperStock ISBN 10: 1-935966-84-7 ISBN 13: 978-1-935966-84-5 Published by Bridgepoint Education, Inc., 13500 Evening Creek Drive North, Suite 600, San Diego, CA 92128 Copyright © 2012, Bridgepoint Education, Inc. All rights reserved. GRANT OF PERMISSION TO PRINT: Bridgepoint Education, Inc., the copyright owner of this material, hereby grants the holder of this publication the right to print these materials for personal use. The holder of this material may print the materials herein for personal use only. Any print, reprint, reproduction, or distribution of these materials for commercial use without the express written consent of Bridgepoint Education, Inc. constitutes a violation of the Copyright Law of the United States of 1976 (P.L. 94-553). www.bridgepointeducation.com I content.ashford.edu lan66845_00_fm_i-xiv.indd 2 4/20/12 2:36 PM Acknowledgments xi Preface xiii Chapter 1 Applications of Psychology 1 Introduction 2 1.1 A Quick Refresher: The Scientific Method 4 Assumptions of Science 6 Cautions About Science 7 1.2 Thinking Like a Scientist 8 Critical Thinking Skills 8 Scientific Terms 10 1.3 How to Scientifically Evaluate a Theory 11 Intersubjective Testability/Falsifiability 11 Internal Consistency 11 Subsumptive Power 11 Parsimony 12 Communicability 13 Heuristic Value 13 Modifiability 14 1.4 Planning for an Applied Project 16 Generating Ideas 16 Applying Psychology 18 Chapter Summary 18 Concept Check 19 Questions for Critical Thinking 20 Key Terms to Remember 20 Web Resources 21 Contents lan66845_00_fm_i-xiv.indd 3 4/20/12 2:36 PM CONTENTS Chapter 2 Practical Matters for Psychology Projects 23 Introduction 24 2.1 Writing Scientifically: A Brief Primer 27 The Major Sections of an APA Research Paper 27 Introduction 27 Method 30 Results 31 Discussion 33 Attention to Detail: Title Page, Abstract Tables, Figures 34 Dissecting the Journal Article Reference 36 Managing Citations 39 Preparation Instructions 41 Common Problems to Avoid 42 A Note About Plagiarism 44 2.2 Ethical Concerns 50 The Ethics of Research with Humans and Animals 50 The IRB and the Role of Informed Consent 52 Anonymity, Confidentiality, and Debriefing 52 Rights of Research Participants 54 Responsibilities of Research Participants 55 Chapter Summary 57 Concept Check 58 Questions for Critical Thinking 59 Key Terms to Remember 59 Web Resources 61 Chapter 3 Between and Within Groups Research Designs 63 Introduction 64 3.1 The Basic Components of Research Designs 65 Pretest-Posttest or Posttest Only 66 One Independent Variable or a Factorial Design 67 Between or Within Groups Design 67 Randomization, Matching, or Blocking 68 3.2 Between Groups Designs 69 lan66845_00_fm_i-xiv.indd 4 4/20/12 2:36 PM CONTENTS 3.3 Participant Selection Challenges in Between Groups Designs 74 Special Situations in Between Groups Designs 75 The Use of Placebos and Double-Blind Experiments 77 3.4 Within Groups Designs 78 Mixed Designs 79 Split-Plot Designs 80 Repeated Measures Designs 81 Participant Assignment: Matching or Blocking 82 3.5 Special Issues Using Within Groups Designs 87 Floor and Ceiling Effects 87 Carryover Effects, Order Effects, and Counterbalancing 89 3.6 Limitations of Experiments 91 Chapter Summary 93 Concept Check 94 Questions for Critical Thinking 95 Key Terms to Remember 96 Web Resources 98 Chapter 4 Quasi-Experimental Designs 101 Introduction 102 4.1 Quasi-Experimental Design Types 104 Nonequivalent Control Groups 105 Time Series Design 105 Cohort Designs and Panel Studies 108 Regression Discontinuity Designs 110 Program Evaluation 112 4.2 Observational Designs 117 Case Studies 118 Naturalistic Observation 119 4.3 Archival Research 121 Content Analysis 122 Meta-Analysis 123 Chapter Summary 126 Concept Check 128 Questions for Critical Thinking 129 Key Terms to Remember 129 Web Resources 130 lan66845_00_fm_i-xiv.indd 5 4/20/12 2:36 PM CONTENTS Chapter 5 Single Subject Designs 131 Introduction 132 5.1 Reversal/Withdrawal Designs 135 Establishing Stable Baselines 137 ABA: The Withdrawal of Treatment 138 ABAB: Repeating Treatments 140 5.2 The Multiple Baseline Approach 143 5.3 The Changing Criterion Design 145 5.4 Data Analysis and Evaluating Change 147 5.5 Applications and Limitations of Single-Subject Designs 148 Chapter Summary 152 Concept Check 153 Questions for Critical Thinking 154 Key Terms to Remember 155 Web Resources 156 Chapter 6 Survey and Questionnaire Research 157 Introduction 158 6.1 Sampling the Population 160 Probability Sampling 160 Simple Random Sampling 161 Systematic Random Sampling 161 Stratified Sampling 161 Cluster Sampling 161 Multistage Sampling 162 Nonprobability Sampling 162 Convenience Sampling 162 Quota Sampling 162 Snowball Sampling 163 Volunteer Sample 163 6.2 Survey Research Methodologies 163 Interviews 163 Telephone Research 164 Mail Surveys 165 Internet Surveys 165 lan66845_00_fm_i-xiv.indd 6 4/20/12 2:36 PM 6.3 Comparisons of Methodologies 167 6.4 Designs for Survey Research 168 Cross-Sectional Survey Designs 168 Longitudinal Survey Designs 168 Cohort and Panel Survey Designs 169 6.5 Scaling Methods 172 Dichotomous Scales 173 Likert Scales 173 Thurstone Scale and Guttman Scale 175 Semantic Differential Scales 175 Other Types of Scales 177 6.6 Analysis of Survey Data 179 Types of Errors 179 Data Handling Issues 180 Data Analysis Approaches 181 6.7 Quick Tips for Survey Item Construction 183 Chapter Summary 187 Concept Check 187 Questions for Critical Thinking 188 Key Terms to Remember 188 Web Resources 190 Chapter 7 Key Concepts: Observation and Measurement 191 Introduction 192 7.1 Variables: Independent, Dependent, and More 193 7.2 Operational Definitions and Related Ideas 197 7.3 The Measurement Process 198 Reliability 199 Test-Retest Reliability 199 Parallel Forms/Alternate Forms Reliability 199 Internal Consistency Reliability 200 Interrater/Interobserver Reliability 200 Validity 201 7.4 Scales of Measurement and Statistic Selection 206 Nominal Scales 206 Ordinal Scales 207 lan66845_00_fm_i-xiv.indd 7 4/20/12 2:36 PM CONTENTS Interval Scales 208 Ratio Scales 209 7.5 Graphing Your Results 211 7.6 Procedural Matters: Experimental Pitfalls and Precautions 213 Confounds 213 Artifacts 214 Physical Setting 214 Within Subjects 215 Demand Characteristics 216 Experimenter Expectancy 217 Pilot Testing Your Study 217 Manipulation Checks and the Post-Experiment Interviews 218 Data Collection and Storage 219 7.7 Causality and Drawing Conclusions from Evidence 222 7.8 Proving Versus Disproving in Psychology 224 Chapter Summary 225 Concept Check 225 Questions for Critical Thinking 226 Key Terms to Remember 226 Web Resources 228 Chapter 8 Applying Psychology: To Workplace, to Life 229 Introduction 230 8.1 Doing Psychology 231 Professionalization in Psychology: Research, Conferences, Publications 232 Local, Regional, and National Opportunities 233 8.2 The Benefits of Undergraduate Research 236 8.3 Key Organizations 238 8.4 Pursuing Graduate Work in Psychology 240 8.5 Finding a Job with Your Psychology Degree 244 Exploring Career Options 244 Transition Tips for Success 248 8.6 What Do You Want? What Will You Do? 249 8.7 A Final Note 252 lan66845_00_fm_i-xiv.indd 8 4/20/12 2:36 PM CONTENTS Chapter Summary 254 Concept Check 255 Questions for Critical Thinking 256 Key Terms to Remember 256 Web Resources 257 Glossary 259 References 271 lan66845_00_fm_i-xiv.indd 9 4/20/12 2:36 PM CONTENTS lan66845_00_fm_i-xiv.indd 10 4/20/12 3:30 PM Clearly efforts such as this do not occur in a vacuum, nor are they ever the result of a sin- gular author working alone. Many individuals share in the credit of bring this resource to fruition. At Bridgepoint Education, I am so thankful for the faith and trust of Mireille Yanow in leading the project and allowing me to contribute this resource designed specifi – cally to benefit the students of Ashford University. Although a bit of a weather snob J, Mireille has been a pleasure to work with and work for. The force is strong in her Bridge – point family, for I also had the pleasure and good fortune to benefit from the hard work of Carrie Brandt, Kristle Maglunob, Nicole Sanchez-Sullivan, and Lindsey Messner. I very much appreciate all of the encouragement and kind words received throughout the devel – opment of this resource. I also want to acknowledge the help and contributions of Saman – tha Gagnon, a former student and now graduate of Boise State University with her bach – elor ’s degree in psychology. Sam was extremely helpful in some of the fact-checking tasks and preparation of key resources such as end-of-chapter glossaries and annotated website listings. With the help and support of this impressive cast of characters, it is my pleasure to provide this resource to you in hope that it will further your understanding and under – graduate education in psychology. I would also like to thank the following peer reviewers for their feedback and guidance: Stephen R. Burgess, Southwestern Oklahoma State University Marion Burke, University of the Rockies, Ashford University Brett Deacon, University of Wyoming Philip A. Gable, University of Alabama Richard L. Moyer, Ashford University Melinda Spohn, Ashford University Adena Young, Missouri State University Acknowledgments lan66845_00_fm_i-xiv.indd 11 4/20/12 2:36 PM lan66845_00_fm_i-xiv.indd 12 4/20/12 2:36 PM Psychology is an active social science. What I mean by that is that psychologists and those trained in psychology “do” things—they act in a certain fashion based on their training. As you will read throughout the text, psychologists think about the world in a very spe- cific manner; a future employer would like to see someone graduating with a bachelor ’s degree in psychology be able to demonstrate the ability to think like a psyc hologist. This text is specifically designed to help you apply the principles of psycho logical research methods to an applied project of interest, whether that be designing a study in a tradi- tional research setting or one based on field observation. Although your coursework may take you only so far with regard to conceptualizing a research project, the resource mate – rials here can take you further into the process, including details about research design, data collection, ethical research, drawing conclusions, writing in APA style and format, and so much more. For a taste of what is in store, I highly recommend that you glance at the table of contents to see the details awaiting you. In thinking about this course and an applied project that is in your future, let me explain why this course (and the content provided here) truly is important. Psychology is an empirical, research-based science. And to be research-based, there must be rules, policies, procedures, techniques, etc.—let’s just call them methods —that are abided by those in any profession. The core materials here provide the foundation of research methods in psy – chology. By utilizing and conquering these methods, you will be qualified with the skills and abilities to ask and answer key questions of interest well beyond the conclusion of this course. Think about the types of evidence that you might experience in the stereotypi – cal “law and order” courtroom—hearsay, eyewitness evidence, forensic evidence, direct testimony, cross-examination, archival precedents, and so on. The legal system has rules for the presentation and relative value of evidence, and there are different sets of rules for drawing conclusions from evidence (beyond the preponderance of evidence versus beyond reasonable doubt). Psychology and the social sciences also have rules of evidence, called research methods. In working on an applied project you will get the chance to dem – onstrate your competency in understanding and applying these research methods to a traditional or field research situation. To help you understand the materials presented in the text, a number of features or enhancements have been added to it. As a Capstone course, one goal is to prepare you Preface lan66845_00_fm_i-xiv.indd 13 4/20/12 2:36 PM PREFACE for the transition from college to career (or from college back to career). This launch is facilitated by a number of approaches, such as • Voices from the Workplace. Because so many psychology undergraduates enter the workforce, this feature is provided at the start of each chapter to demonstrate the variety of skills needed and career paths available to those with a bachelor ’s degree in psychology. • Case Study Features. Each chapter has a case study that explores one topic in more depth that most of the research presented throughout the text. This allows for some deeper understanding and study of an important topic from multiple perspectives. • Classic Studies. Santayana said, “Those who are ignorant of the past are doomed to repeat it.” With this in mind, each chapter takes a minor detour (or stroll) down memory lane, providing an opportunity to dig into one of the many classic studies in psychology. This allows a demonstration of how past studies continue to affect our thinking about current research, as well as provide important con – text and framing so that we do not neglect or ignore our disciplinary past. As you would expect with any text, other features are highlighted within each chapter, including chapter learning objectives, critical thinking questions to en courage deeper con- sideration of ides, end-of-chapter glossaries and chapter summaries, and annotated web links. If you want to explore the origins and interconnections behind the ideas presented in text, all the breadcrumbs are there for you to follow. R. Eric Landrum, Ph.D.Boise State University Boise, Idaho lan66845_00_fm_i-xiv.indd 14 4/20/12 2:36 PM 1 Applications of Psychology Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • describe the scientific method and the assumptions and values of science. • articulate the characteristics of critical thinking and exhibit critical thinking behavior. • understand the dimensions upon which theories are evaluated. • generate ideas and locate information relevant to an applied project in psychology. Comstock/Thinkstock lan66845_01_c01_p001-022.indd 1 4/20/12 2:43 PM CHAPTER 1 Introduction Introduction D o you like roller coasters? Do you like the twists, turns, unexpected outcomes, loops, G-forces, anticipation of the climb to the top, and all the complexity of a 2-minute roller coaster ride? So much planning and detail goes into the experi – ence—there is the slow climb to the top, the rapid acceleration, the mutual joy and fear of coming up off your seat but being held in by the harness. In many ways, the tools of psychology can give you the ability to tell compelling stories, by setting the stage care – fully, revealing the action with its own twists and turns, and then the inevitable compari – son to prior stories (and roller coasters). If you are seriously interested in understanding the mysteries of human behavior from an objective and systematic perspective, then the ability to apply research methods in psychology is an essential tool for you to acquire and master. Like any complex tool, it will take time and practice to master, and the ultimate goal of this book is to help you start (or continue) your journey on this exciting path. Fundamentally, psychology is an empirical (research-based) science, and if you want to have any understanding of psychology and what psychologists do, you should under – stand research and apply research methods. It’s our set of principles—principles that transcend different specialty areas and training approaches across psychology. Even if you have no desire to be a researcher in the future, you’re going to need to understand research and be a good consumer of psychological research. Some students will say that they only want to “help people” and that they do not want so much background and training in research. However, the cutting-edge areas of clinical and counseling psychol – ogy formulate new therapeutic approaches on evidence-based treatment (Kazdin, 2008). Think of it this way: If you want to help people, it would be nice to have both confidence and evidence that you are helping. Having a chance to demonstrate your knowledge and skills prior to graduation is an important opportunity for both you and your institution. Voices from the Workplace Your name: Courtnee R. Your age: 29 Your gender: Female Your primary job title: Special Investigator Your current employer: Florida Farm Bureau Insurance Co. How long have you been employed in your present position? 3 1/2 years What year did you graduate with your bachelor’s degree in psychology? 2001 Describe your major job duties and responsibilities. I investigate suspicious/questionable and fraudulent claims submitted to my company on behalf of the insured party or claimant party. I handle varying forms of issues that range from auto accidents in which someone is claiming an injury that is questionable in nature or potentially pre-existing that they are claiming occurred as a result of the current accident, to home invasions resulting in damage to the property or theft of contents. I also investigate staged auto accident rings, fraudulent or (continued) lan66845_01_c01_p001-022.indd 2 4/20/12 2:43 PM CHAPTER 1 Introduction phony billing companies, stolen autos, and catastrophe claims resulting from tornadoes and/or hur- ricanes. I also conduct extensive background investigations and interviews. What elements of your undergraduate training in psychology do you use in your work? On a daily basis I am analyzing background data on people, medical documents, police reports, prior interviews, etc. Due to the finite and specific nature of the documents and information I am looking at, I have to have concentrated attention to detail. Many of my undergraduate psychology classes empha – sized training in this—giving me the skills I now utilize on a daily basis. Additionally, I conduct many interviews wherein I have to have a thorough understanding of human nature and behavior in order to obtain the information I need or am trying to uncover. What do you like most about your job? I love the independence and flexibility my job allows. My manager trusts and expects that I perform my job well, and without handholding. I’ve earned that privilege over time and now reap the benefits of leaving at noon if I am done for the day. What do you like least about your job? No one likes insurance companies, so I am constantly working against the mindset that it’s okay to steal a little or lie to an insurance company to get what you need. After all, insurance companies have tons of money, right. . . . Beyond your bachelor’s degree, what additional education and/or specialized training have you received? I have no other degrees besides my BS in Psychology. However, I have numerous specialized classes under my belt that I have taken in the course of my previous job and my current one. Prior to my cur – rent position, I worked as a Crime Scene Investigator with a police department. I received extensive specialized training with this position. What is the compensation package for an entry-level position in your occupation? My company is smaller, so they have great benefits and perks. I drive a company car that comes with free gas (personal or otherwise), insurance, and all maintenance. I have a free cell phone, and an unbelievable 401k/health insurance package. What are the key skills necessary for you to succeed in your career? Being able to work independently is the biggest item. Setting your own schedule, staying on task to get your cases taken care of without needing to be told or checked in on. I also work primarily with men, and as a female I have to have a lot of confidence and gumption to work in my environment. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? I didn’t belong to any clubs in college (although I probably should have). I worked all through college at the Attorney General’s Office as a legal assistant/paralegal and this has aided me so much in my career. So many people hurry up and get out of school in 4 years or less, and don’t work or do any outside activities during. If, once you graduate, all you have is your degree and no other experience to draw from, you are going to be hard-pressed to sell yourself to potential employers. They want to know that you can carry on a conversation with people outside the classroom and have real-life experience and skills other than sitting in a classroom. I can’t stress enough how important and beneficial it is to get to know your professors. If you just show up to class and sit in the back of the room, you will just be another name out of hundreds to that professor. Get to know him or her! Whether you go on to grad school, or start working, you will be required to have letters of recommendation. If no one knows who you are, he or she can’t write anything about you in a letter. Get to know your professors! Voices from the Workplace (continued) (continued) lan66845_01_c01_p001-022.indd 3 4/20/12 2:43 PM CHAPTER 1 Section 1.1 A Quick Refresher: The Scientific Method 1.1 A Quick Refresher: The Scientific Method Science is not the only way by which to understand the world—we use other approaches such as logic and intuition. But a scientific approach allows for conclusions to be drawn with a certain amount of confidence in the accuracy of those conclusions. The values of science can be summarized as follows: • Science places high value on theories that have the largest explanatory power. If you have a theory that can explain the outcomes of a great many hypotheses, that theory is highly useful. • Science values predictive power. The more we can predict future behavior based on scientific data, the more useful that data will be. • Science values fecundity —that is, fruitfulness (the generation of new ideas). If a theory can help us generate new ideas about the world we live in, then that theory has value. Sometimes this characteristic is known as the heuristic value of a theory. • Science values open-mindedness. Critical thinking and open mindedness are essential tools for success in science (more on critical thinking later in this chapter). • Science values parsimony . When all else is held constant, scientists prefer sim – pler theories to more complex theories. • Scientists require logical thinking in their explanations. By making our processes and results public, science must withstand public scrutiny and make cogent arguments for their conclusions. • Scientists value skepticism . Although anecdotal evidence may be memorable (and often entertaining), scientists want evidence in such a way as to either sup – port or refute a claim (Smith & Scharmann, 1999). • Science is self-correcting. As an undergraduate, do you wish you had done anything differently? If so, what? Slept a little more, and tried to get a little better at my math skills. What advice would you give to someone who was thinking about entering the field you are in? Basically everything I’ve outlined in the other questions, but be exceptional. Stand out. Make a path for yourself with dedication, smarts, and charisma. If you were choosing a career and occupation all over again, what (if anything) would you do differently? I’m not sure I would change a thing. My life has been interesting and filled with adventure and happi – ness. Maybe taken a few more statistics classes. . . . Copyright . 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. Voices from the Workplace (continued) lan66845_01_c01_p001-022.indd 4 4/20/12 2:43 PM CHAPTER 1 Section 1.1 A Quick Refresher: The Scientific Method By sharing the methodology of research studies, erroneous find- ings and mistakes can be identi – fied and flaws in the literature can be corrected. Note the phrase “research studies”—it is a very broad term that applies when scientific methods are used to advance our understanding. An experiment is a specific type of research study, and naturalistic observation is another approach to conducting a research study. In due time (and in future chap – ters), you’ll explore these differ – ent type of research studies in further detail. Understanding the structure of how (and why) we do research will help you to become a better (and more informed) researcher. So what is the scientific method ? The word “science” comes from the Latin word scientia , meaning knowledge. But the word science has at least two different implications. First, science is a method of studying the world around us. Remember learning the five-step scientific method in a grade school science class? Those steps are listed on the left side of Figure 1.1. That sci – entific method that your grade school science teacher taught you is the same scientific method that is used today in chemistry, physics, biology, and psychology. So when psy – chologists write the introduction in a research paper, they are describing the problem to be solved by the research as well as formulating a hypothesis to be tested. More parallels are available in Figure 1.1. Generate a problem Introduction Form a hypothesis Conduct an experiment Method Collect data Results Draw a conclusion Discussion The five steps of the scientific method that many of us learned in grade school are the same steps scientists use in real-world applications. Figure 1.1: The scientific method and psychology When the methodology of research studies is shared it becomes easier to find mistakes, promoting learning and improving methods. Photodisc/Thinkstock lan66845_01_c01_p001-022.indd 5 4/20/12 2:43 PM CHAPTER 1 Section 1.1 A Quick Refresher: The Scientific Method The second implication for the word science involves the production of knowledge. This may seem redundant, but science produces scientific knowledge. Intuitively, you already know this. Haven’t you seen a TV advertisement or a news broadcast or magazine arti – cle or an Internet blog that mentions a non-scientific survey? That disclaimer is added because scientific survey results differ in method and outcome from a non-scientific sur – vey. Why is that distinction important? Because knowledge gained through a scientific approach has certain characteristics that are different from knowledge gathered in other ways. For example, consensus is the basis for scientific knowledge. We make our results known publicly through scholarly articles, books, websites, conference presentations, and so on. In addition, scientific conclusions are tentative, (hopefully) based on the best data available. So data that are gathered using the scientific method produce scientific informa – tion that differs from that gathered by other methods. Science will not always be the best method for gaining information. For now, it’s important to remember that science is both a method and a product. Assumptions of Science Two very commonly mentioned assumptions of science are determinism (or lawfulness of nature) and finite causation (Maxwell & Delaney, 2004; Underwood, 1957). To this traditional listing we can add a third assumption of science—empirical evidence. The notion of determinism is fundamental for scientists to make any sense of the world we live in. Determinism posits that events that occur are lawful events; that is, they are predictable. Determinism goes a bit further to state that if all the causes were known for an event, that event would be completely predictable. Maxwell and Delaney (2004) summarize this view in three parts: (a) nature is understandable, (b) nature is uniform, and (c) events in nature have specific causes. If you think about how this applies to human behavior, it makes sense. When you go to the doctor for an ailment and receive a prescription, you expect by taking the prescription that the ailment will be resolved. You already believe in cause and effect. You believe that if you take your medications, you will get better. Perhaps you also believe that going to college will result in a better life for you. Although free will certainly has a role to play in our lives (see Baumeister, 2008), scientists depend on this idea of determinism to help make sense of the world. If there weren’t causes and effects, then events and behaviors would be random and not very predictable at all. This underlying belief in determinism is related to the second assumption of science— finite causation . Not only do scientists believe in cause and effect, but for practical rea – sons, there are a finite (limited) number of causes for any effect or event, and these causes are discoverable and understandable (Maxwell & Delaney, 2004; Underwood, 1957). “Sci – ence would be almost a hopeless undertaking if nature were so constituted that every – thing in it influenced everything else” (Underwood, 1957, p. 6). Of course the challenge to scientists is to identify which causes are leading to the effects and under what conditions those cause-and-effect relationships occur (Maxwell & Delaney, 2004). The third underlying assumption of science is related to the type of knowledge it pro – duces—empirical evidence . Science, at its best, produces knowledge that is both reli – able and public. Reliability occurs in the form of self-correction, because consensus is lan66845_01_c01_p001-022.indd 6 4/20/12 2:43 PM CHAPTER 1 Section 1.1 A Quick Refresher: The Scientific Method the foundation for scientific knowledge. In science, nothing is self-evident at first, and scientific claims must be supported. Knowledge is acquired through empirical (research- based) experience, and the claims of science are tentative, based on the best information currently available. Cautions About Science A theme expressed earlier in this chapter is that although science will be the primary lens used throughout this book, science is not the “magic bullet” that solves all problems and provides all-knowing insight. Science and the scientific approach clearly have cautions and limitations that you should know about so that you know “the rest of the story.” First, as scientists we deal in probabilities. Psychology is empirical, and psychologists gather data and analyze data and draw conclusions from the data. These conclusions should be thought of as tentative. Science isn’t perfect, and we acknowledge that the findings we settle on have a high probability of being correct, but with no guarantee. Second, we are limited to what we can understand about our world based on the physical equipment we possess. We rely on our eyes to see, our ears to hear, and so on. We are only as accurate and precise as our sensory system and the tools we use to amplify our sensory perception. Thus, for the clearest view of how a neuron is shaped, we have to rely on the best microscopes available. There are light wavelengths in the electromagnetic spectrum that we cannot “see” because our eyes are unable to do so. There are sound waves that we cannot “hear” because the frequency of those sounds is too high or too low for the capa – bilities of our auditory system. Third, science is limited to the investigation of public knowledge—the observable and quantifiable. Thus, science cannot answer questions about value (Fancher, 2004)—for example, which is more valuable, liberty, or the pursuit of happiness? These are opinion questions that each person must answer for him- or herself—science is not able to provide the right answer. Of course, scientists can certainly poll individuals and determine what prevailing beliefs are, but the most common answer would not necessarily be labeled the right answer. Related to questions of value, science is ill-equipped to answer questions of morality. Questions like “Is capital punishment wrong?” or “Does life begin at concep – tion or birth?” are important, but they are not good questions for science if you want the “right” answer. Science doesn’t tell us what is right or wrong; we decide that for ourselves (Fancher, 2004). Finally, science is also limited to study of the natural and physical world—said another way, science cannot help us to answer questions about the supernatural (e.g., the occult, clairvoyance, witchcraft). By definition, scientists need to study what exists in nature. It is not that value-based, moral, and ethical questions are uninteresting, but the methods and approach of science are not well-suited for such judgments. The scientific approach has its strengths and its limitations. lan66845_01_c01_p001-022.indd 7 4/20/12 2:43 PM CHAPTER 1 Section 1.2 Thinking Like a Scientist 1.2 Thinking Like a Scientist C ritical thinking is defined as a set of strate- gies designed to make us better consumers of information (Wade, 1990). Although you may not become a scientist, you are and will con – tinue to be a consumer of information. It’s obvi – ous that you consume information from profes – sors and textbooks, but you are also a consumer of information at the supermarket, on the Inter – net, in front of the television, watching a YouTube video, and at the car dealership. Critical think – ing strategies outline an approach to consuming information that ideally helps a person be less gullible. Critical thinking applies to all aspects of decision making. A more specific version of critical thinking, the evaluation of theories, is pre – sented later with detailed guidelines for judging a theory on its merits. Being critical in this sense does not mean criticizing; it means carefully and meticulously examining the points and perspec – tives of information presented to you. Critical Thinking Skills The following critical thinking strategies will help you become a better consumer of information. 1. Be skeptical, ask questions, and be willing to wonder . A critical thinker is not duped into believing anything he or she is told; he or she respectfully questions the sources of information and asks questions when they are confronted with details that do not make sense. Given your interest in human behavior and psychol – ogy, asking “what if” questions should come naturally. For example, think about questions like (a) What would happen if I started studying the night before the test? (b) What would happen if I treated my mate the way I want to be treated? or (c) What would happen if politicians delivered on their promises? How would you like to be able to apply psychology? Your natural curiosity about behavior combined with practice and applied research methods skills can provide you with the framework and tools to begin to answer, scientifically, the questions of interest to you. 2. Define the problem, and examine the definition of terms . Defining the problem is one of the first key steps of the scientific method. If we want to understand or predict some sort of behavior, we need to be able to describe and define what that behav – ior is. In psychology, we rely on operational definitions to help us know what we are precisely measuring. 3. Examine the evidence . A critical thinker looks for and examines the evidence on an issue. In psychology, we tend to rely on empirical evidence, but other types of evidence can be useful at times. For example, anecdotal evidence (evidence based on a personal story or experience) can sometimes be useful in thinking about Critical thinking is a set of strategies that help us to be better consumers and disseminators of information. When was the last time you used your critical thinking skills? Creatas/Thinkstock lan66845_01_c01_p001-022.indd 8 4/20/12 2:43 PM CHAPTER 1 Section 1.2 Thinking Like a Scientist problems to solve or generating plausible hypotheses, but anecdotal evidence is generally not considered as scientific or empirical support for a hypothesis. In examining the evidence of others, we look for the standards of science to be met. 4. Be cautious in drawing conclusions from evidence. Two scientists looking at the same evidence might draw different conclusions. Often evidence is preliminary in nature and conclusions should be delayed until more evidence is available. Fur – thermore, some types of evidence only support some types of conclusions. For example, evidence from correlational studies suggests a statistical relationship between variables, and not a causal relationship . In other words, cause-and- effect conclusions are rarely drawn from correlational evidence. There is usually a positive correlation between height and weight, but it is awkward and incorrect to conclude that height causes weight or that weight causes height. Just because two variables are significantly correlated does not mean that values of one vari – able cause the values of the other variable to change. Age is typically correlated with net worth, but we know of young millionaires and of elderly who live in poverty. As scientists, we want evidence, but we need to be cautious about the types of conclusions we draw based on the type of evidence we have. 5. Consider alternative explanations for research evidence . One of our primary tasks is to consider alternative explanations in the interpretation of research findings. In fact, most researchers consider alternative explanations before conducting stud – ies—in later chapters you’ll learn more details about how to rule out alternative explanations. The psychologist is constantly looking for alternative explanations, ideally before the research is conducted; in reality, though, sometimes we specu – late about alternative explanations after a study is complete. 6. Examine biases and assumptions . The critical thinker must examine his or her own biases and assumptions as well as the biases and assumptions of others. Informa – tion comes to us from a source, and this source almost always has a certain slant, angle, or spin. That is, the messages we receive are sent for a purpose, and often that purpose is to influence our thinking, change our attitudes, or influence our behavior. It’s also important to remember that we are not always impartial, even when we think we are. 7. Avoid emotional reasoning as a substitute for reasoning . We are emotional beings— there is no denying that. Emotions can sometimes cloud and disturb our deci – sion-making skills, and the available evidence is not often utilized fully when emotions are peaking. Sometimes we get caught in the trap of “since I feel this way, it must be true.” We may focus too much on one element of the situation, without gaining the perspective to understand the multiple sides of an issue. Feelings and emotions are important, but they should not be substituted for care – ful examination of the evidence. 8. Try not to oversimplify or overgeneralize . We live in a complicated world and it is rare that we can condense an important issue down to a convenient (yet accurate) generalization. Oversimplifications and overgeneralizations are mental shortcuts that we all make to help us organize and understand the world, and to take a break from critical thinking. Some of our prejudices and stereotypes are based on unfair overgeneralizations. Living in a complicated society dictates that compli – cated solutions must be generated; simplistic either-or thinking usually doesn’t work well in the long run. Although there are times when we simplify and gener – alize for ourselves and others, trying to force the complexity of the world we live in into neat, organized cubbyholes leaves us with an oversimplified view of the world that is not that accurate. lan66845_01_c01_p001-022.indd 9 4/20/12 2:43 PM CHAPTER 1 Section 1.2 Thinking Like a Scientist 9. Tolerate uncertainty . Science is based on tentative answers formed from the best evidence available at the time. Key in on the phrase “at the time.” Given the reli – ance on evidence, the critical thinker from time to time may change position on a topic due to personal experience and the emergence of new evidence. In the sci – entific community, new evidence emerges frequently. When evidence and beliefs change, uncertainty is present. Consider the information we receive about our health. It is common, for example, to read a study that says oat bran is good for you and that it lowers your cholesterol, but six months later read another study that suggests that oat bran has no effect on cholesterol. What do you believe? The critical thinker must examine the evidence looking for alternative explanations, and if a different conclusion is warranted by the evidence, change his or her opinion. Sometimes this uncertainty is what causes anxiety and fear. Our ability to tolerate uncertainty suggests being open-minded to change yet still applying the rigorous standards of critical thinking. Scientific Terms The four terms listed here are precise terms when used in a scientific context, which varies from our more common usage of these terms (just as the terms “significant” and “power” have special meaning in an experimental context). A fact is the result of careful observa – tion; a fact does not offer an explanation of what has happened, but rather offers a descrip – tion of the event or behavior. A hypothesis is an educated guess that attempts to explain the facts that result from scientific studies (our “data”). A hypothesis can be tested utilizing research methods, and the con – clusion drawn from a hypoth – esis test is that the hypothesis is either supported or refuted. We don’t prove hypotheses; a hypothesis can be disproved, however (Carrier, 2001). A theory is an attempt to explain facts that are often tested as research hypotheses. That is, a theory tries to predict the facts or data that exist, often as a sum – mary of hypotheses that have been repeatedly tested. This the – ory continues to be held by sci – entists until contrary evidence is presented—in others words, theories can be disproved. The theory provides an inference about the data (facts) observed and the explanations as to why the events occurred. A law is a generalization for a collec – tion of facts, but without explanation (the explaining is left up to the theories). Scientific laws are identified when no exceptions have been found to the law; scientific laws explain what has happened, but it is scientific theories that attempt to explain why something happened (Carrier, 2001). A theory tries to explain facts often tested as a research hypothesis. Einstein’s Theory of Relativity is an example of how theory and research can lead to a scientific law. Hemera/Thinkstock lan66845_01_c01_p001-022.indd 10 4/20/12 2:43 PM CHAPTER 1 Section 1.3 How to Scientifically Evaluate a Theory 1.3 How to Scientifically Evaluate a Theory I n psychology, theories are commonplace. Theories serve a number of useful functions in thinking about meaningful patterns of behavior. First, theories help us to organize the results of numerous studies. Tens of thousands of journal articles are published each year, and these articles contribute to our knowledge of human behavior. Theories help to organize these findings into more coherent groups of ideas. Second, theories often stimulate others to do research. A psychologist may set out to find evidence to either sup – port a theory or refute it, but the theory may provide some motivation in this process. Morris (1985) presented seven criteria to be used in evaluating theories, providing some basis for distinguishing between good theories and not-so-good theories. In addition, other advantages emerge when theories are used in psychology. Intersubjective Testability/Falsifiability Theories possess intersubjective testability when they generate hypotheses that are test – able from an empirical standpoint. That is, if a theory is falsifiable, evidence can be gath – ered to negate or deny the hypothesis. Theories should provide ideas that are testable, meaning the theory can be tested and found to be supported or not. Internal Consistency Often theories are complicated entities that have a number of stipulations to be used accurately. A theory that is internally consistent does not have contradictions maintained within the body of the theory. That is, predictions made by one part of the theory should not be contradicted by other parts of the theory. There should be minimal con – tradictions in the expected outcomes when the theory is applied in a variety of situations. Subsumptive Power A good theory has subsumptive power , which means it can account for the results of prior studies while offering a theoretical frame – work. Later theories need to be able to subsume (absorb, explain, account for) the findings of earlier research studies. If a theory lacks this explanatory power, the benefits of the theory may be limited. New research that emerges must be able to place the context of the past into the present. For example, if you want to pro – pose a new theory of cognitive development in children, you cannot just ask the scientific com – munity to discard 100-plus years of work in Theories should have internal consistency, which means that new research findings should further support the theory and not contradict findings from past research. BananaStock/Thinkstock lan66845_01_c01_p001-022.indd 11 4/20/12 2:43 PM CHAPTER 1 Section 1.3 How to Scientifically Evaluate a Theory psychology (and other disciplines)—you need to fit your new theory into our existing framework of knowledge. If the new theory of cognitive development were to displace Piaget’s theories of cognitive development, you would have to show how your theory explains existing data better, or you might need to point out flaws in Piaget’s theory that your new theory avoids. In other words, new theories have to fit old data. Parsimony To say that a theory is parsimonious means that it is simple yet complete. When a theory is said to be simple (not simplistic), it explains the events or phenomena of interest in the least complicated terms. Good theories are parsimonious and they make sense. You may have heard the story about Clever Hans, a horse that could supposedly answer math problems, spell, solve word problems, etc. One theory for the horse’s ability was that it was an extraor – dinary animal capable of human-like problem-solving ability. Another more parsimonious (and correct) theory was that Clever Hans was incredibly sensitive to the nonverbal head movements of the people asking questions. For more on the story of Clever Hans, see the fol – lowing case study. Simple, parsimonious theories are superior to more complicated theories. Case Study: The Story of Clever Hans Sometime around 1902 a retired German schoolmas- ter, Herr von Osten, placed an advertisement in a couple of newspapers to sell a horse that could identify 10 colors and knew the 4 rules of arithmetic. Nothing happened when this ad was first placed, but when the ad was run again about a year later, a brigadier general in the German army noticed, and expressed an inter – est in the horse, named Clever Hans ( der kluge Hans in German). The general’s attention brought wider atten – tion to Herr von Osten and Clever Hans. The phenomenon of Clever Hans was well studied at the time, and published accounts exist of this story (Fernald, 1984; Sanford, 1914). This was no hoax. People from around the countryside near Berlin gathered to see the horse complete remarkable arithmetic tasks. No money was taken from sightse – ers, and von Osten would not put the horse in a public exhibit. Clever Hans answered questions not only for von Osten, but for many others, including reputable individuals from neighboring communi – ties and learned individuals from the University of Berlin. What types of problems would Clever Hans solve? When told to count to a number, Clever Hans tapped his forefoot that number of times. When given math problems, he tapped out his answer, which was often (but not always) correct. Using blackboards, Clever Hans would touch his nose to the answer to the question asked. Remember, these were not tricks—the horse was not receiving intentional signals, and in fact von Osten had spent over four years training the horse. As word spread about the prowess of Clever Hans, he gained more attention, eventually eliciting attention from faculty members from the University of Berlin. So was the performance of Clever Hans a stunt? Initial investigations from learned individuals such as Dr. Pfungst and Professor Stumpf of the University of Berlin could detect no trickery, and it seemed that Clever Hans was indeed clever. But after multiple sessions of carefully crafted test – ing, utilizing some of the same research methods discussed throughout this book, Pfungst deter – mined that “if the questioner knew, Hans knew; if the questioner did not know, Hans did not know” Mary Evans Picture Library/Everett Collection lan66845_01_c01_p001-022.indd 12 4/20/12 2:43 PM CHAPTER 1 Section 1.3 How to Scientifically Evaluate a Theory Communicability For a theory to be useful it must be communicated in an understandable fashion. This is easier said than done. You may have discovered that much of the time articles from psychology journals are not easy to read. Psychology uses its own jargon, terminology, and style to communicate the results from research studies. This may seem obvious, but if a theory is going to be influential it must be communicated so that others reading it can understand it. If your theory is not very understandable and cannot be communicated effectively to others, how will they benefit from it? Heuristic Value Probably one of the most impor – tant characteristics of a theory is its heuristic value, that is, its abil – ity to motivate others to conduct research on the topic (also called fecundity). Often a theory can do poorly when evaluated against the other criteria discussed in this section, but if a theory has heuris – tic value, then even a poorly sup – ported theory can have value. An example of a theory that motivated a great deal of research was Rush – ton’s (1988) idea that the intelli – gence differences found between African-Americans and Caucasians were due to the genetic inferior – ity of the African-American race. Other researchers (and the public) (Sanford, 1914, p. 10). Continued, careful research with Clever Hans determined that his real gift was his ability to detect faint nonverbal cues. Imagine if we were gathered in a semicircle in a courtyard listening to von Osten or Pfungst, and the questioner were told to ask Clever Hans “What is the cube root of 8?” If the questioner did not know the answer (2), then Hans would not answer correctly either. But when asked for the answer to 2 + 2, you can imagine us in the crowd looking up at Clever Hans and gasping when he tapped his forefoot to 4, and then stopped. Think of how amazed we would be, watching this performance! Clever Hans was adept at detecting the subtle, nonverbal cues given off by the crowd or the questioner, and as soon as he noticed head movements or heard non – verbal cues, he stopped what he was doing (e.g., pointing with his nose, tapping with his foot). Inadvertently, the researchers were causing an experimenter expectancy artifact; Clever Hans was not as intelligent as first believed but rather was sensitive to the nonverbal cues emitted by his question- ers. An artifact describes something that happens during a research study that was not intended to be a part of the research study yet has the capability to influence the outcomes of the study. As research – ers, we want a fair and unbiased test of our hypothesis—we do not want to influence (artifactually) the outcomes of our experiment. Case Study: The Story of Clever Hans (continued) When Rushton proposed that differences between whites and African-Americans were due to genetic inferiority it had an inherent heuristic value because it inspired other researchers to disprove his theory. Lifesize/Thinkstock lan66845_01_c01_p001-022.indd 13 4/20/12 2:43 PM CHAPTER 1 Section 1.3 How to Scientifically Evaluate a Theory were outraged about the theory, and many psychologists began studying the sources of intelligence differences between races. Many alternative explanations were found, such as family structure and environment, socioeconomic status, and cultural bias in intelligence testing. Though Rushton’s original theory was supported by few, the theory was useful in that it inspired others to pursue this line of research; that is, the theory had heuristic value. Sometimes a bad idea or negative beliefs can inspire good work. Modifiability The last component of evaluating a theory is judging its modifiability. For a theory to stand the test of time, it must be flexible and modifiable enough to incorporate the studies of the future as well as the studies of the past (subsumptive power). A theory that is rigid and narrow and based on very situation-specific evidence usually does not last very long. A good theory is able to incorporate past research and anticipate future research. Classic Studies in Psychology: On Memory (1885) by Hermann Ebbinghaus How does one person early in the history of psychological research not only make such a lasting impact but also influence so many aspects of psychology today? How remarkable is it that Hermann Ebbinghaus’s lasting contributions range back to 1885? First, Hermann Ebbinghaus’ classic work (1885) Memory: A Contribution to Experi – mental Psychology (this is the translated title from the original Ger – man) is truly one of the first—if not the first—systematic psychological studies of memory. Second, Ebbinghaus used innovative techniques to study memory. Third, and perhaps most impressively, Ebbinghaus’ methodological techniques were so rigorous that his conclusions from over 120 years ago are still meaningful today. His classic forget – ting curve is often featured in the memory chapter in introductory psychology textbooks. So what was Ebbinghaus’ approach to studying memory that made his results truly stand the test of time? Ebbinghaus began his now-classic series of memory experiments in 1879, and his research lasted into 1880 (Ebbinghaus, 1885). So methodical and precise, Ebbinghaus replicated (repeated) the entire sequence of year-long studies 1883–1884 before publishing his 1885 book, Über Das Gedächtnis (translated from German as “On Memory” or “Concerning Memory”). Continuing with the theme of the time (introspection), Ebbing – haus selected himself as the participant in his memory studies (Plucker, 2007). His inventions for meth- odological rigor are perhaps unparalleled in psychology. Ebbinghaus realized early on that he would not be able to use real words in his memory testing because the associations of these words would be helpful retrieval cues, and not provide an accurate test of memory processes such as forgetting. So Ebbinghaus created nonsense syllables composed of a consonant-vowel-consonant combination that was pronounceable but had no inherent meaning. Ebbinghaus created 2,300 nonsense syllables that he used over the course of his memory research on himself. Another of Ebbinghaus’ lasting contributions is his forgetting curve. Systematically over time he would memorize a list of nonsense syllables to perfection, let a period of time pass, and check to see how much he had forgotten and how much he could remember. Then he would relearn the list to perfec- tion, noting how much time it would take. Relearning the list the second time was quicker Corbis/AP Images (continued) lan66845_01_c01_p001-022.indd 14 4/20/12 2:43 PM CHAPTER 1 Section 1.3 How to Scientifically Evaluate a Theory than the original list learning, and Ebbinghaus called this advantage to learning the list a second time savings. Figure 1.2 shows the Ebbinghaus forgetting curve based on his actual 1885 data. That Ebbinghaus was able to systematically study memory using an empirical approach and sophisti- cated laboratory-like controls is a tribute to his rigor. That his methods and his findings are still cited throughout psychology today is amazing. As one scholar put it (Shakow, 1930, p. 509), Ebbinghaus published high-quality work at the time but did not publish the quantity that other similar scholars published at the time for “. . . the fact that Ebbinghaus probably had no typewriter.” Reflection Questions 1. What characteristics led to the lasting impact (i.e., legacy) of Ebbinghaus’ success? How might leaving a legacy based on work conducted in 1885 be different from leaving a legacy in today’s world? 2. The work of Ebbinghaus is often called the first significant psychological study of memory. How does being the first change the nature of the contribution? Are there advantages to being first in a field? Are there disadvantages to being first in a field? For each question, please describe. 3. Think about your own work flow and how you optimally use (or do not optimally use) technology. Then consider that for Ebbinghaus’ prolific career, he probably did not have access to a type – writer. In thinking about your coursework and past and present employment, describe any work situations where your actual work may have benefited from less technology? How does your use of technology affect the quality of your work? Can you think of work/school situations where the quality of your work might improve if you relied on technology less? Relied on technology more? Re tention (percent) Immediate re call 20 minutes 1 hour 9 hours Elapsed time (days) 10 0 80 60 40 20 2468 101520 25 31 This is the classic Ebbinghaus forgetting curve, which demonstrates that as time passes less of what was originally learned is retained. That is, as time passes, the accuracy of memories decline. Source: Adapted from Ebbinghaus, H. (1885/1913). Memory: A contribution to experimental psychology. Figure 1.2: Ebbinghaus forgetting curve Classic Studies in Psychology: On Memory (1885) by Hermann Ebbinghaus (continued) lan66845_01_c01_p001-022.indd 15 4/20/12 2:43 PM CHAPTER 1 Section 1.4 Planning for an Applied Project 1.4 Planning for an Applied Project D epending on the course structure, you may be given a paper topic to write about, or you may have some leeway in the topic you can select for your research paper. If you are given your topic, well, of course, write about that topic. If you want to do something non-mainstream with the assigned topic, check with your instructor first. You don’t want to go to a lot of trouble to do something you think is unique and fascinat – ing just to find out that the instructor won’t accept it. But what will your research be about if you have some choice in the matter? I have a few suggestions for you. The first rule is, of course, to satisfy the instructor ’s assignment. But beyond that rule, how do you select your topic? First, try to select a topic that you are passionate about. If you had free time in college and could study any topic that you want, what would that be? Perhaps you have a relative who was just diagnosed with Alzheim – er ’s disease, and you’d like to know more about how behavior changes over time with Alzheimer ’s. Do research about that topic. Or perhaps you just went through a painful breakup with your significant other, and you are having a hard time coping. Why not do research about coping skills and how people recover from stressful life events? If you can connect your research paper topic to something in your life that is relevant, then the task of doing all the work that goes into a project might not seem so daunting. Generating Ideas First, find a topic that is relevant to you and that you are passionate about—but remember that you must be open-minded enough as a critical thinker to look for evidence on all sides of an issue, including sides that you do not necessarily agree with. If you cannot put your passions aside to see both sides of an issue, then perhaps that’s not a good topic for a research study. Second, before you submit your idea to your instructor, do a quick search of the existing litera – ture. Be sure to inquire about what databases your institutional library subscribes to. Google Scholar (http://scholar.google.com/) is also a good place to start a literature search. The goal of a litera – ture search is to see what research is already out there. If there is almost nothing about your topic in the published literature, this could make your research more difficult (however, if there is a rela – tively small amount of literature, this is also the case where a single study can make a relatively large contribution). If there are tens of thousands of articles available on your topic, you’ll have to narrow your search so that you can review a manageable number of previous studies. Once you have identified potentially valuable sources When choosing your idea before submitting it to the instructor, do a quick search of available literature at your library or use the library’s existing databases like Google Scholar. Creatas/Thinkstock lan66845_01_c01_p001-022.indd 16 4/20/12 2:43 PM CHAPTER 1 Section 1.4 Planning for an Applied Project of information, see if your journal articles and book chapters are available with an easy mouse-click to download a PDF, or figure out if you will need to access your resources through your library. Some may be available in your campus library, electronically, or available through interlibrary loan. Do your homework on your topic before you fully commit to it, especially if there are tight deadlines for your project. Consult with your teacher or reference librarian for more ideas about databases to search. Third, if you are struggling to find a topic you are passionate about, look around at your life and those in your life to find something interesting. Hopefully you have a keen inter – est in human behavior already, so follow that lead. If you want ideas about what aspects of human behavior to explore, watch people. Spend a couple of hours on a bench in your local mall, and you’ll be exposed to all sorts of interesting behaviors that might be worthy of a research study. Or select a topic that is of personal interest to you. For instance, you may not be too thrilled with the company that provides a television signal to your house. So a relevant research topic might be about identifying the factors that lead to one’s satis – faction with TV services and having people rate the importance of factors like cost, quality of picture, variety of channels available, frequency of outages, and value for the service. You can make your research into an applied question of interest to you, just as the TV example. My own line of research tends to follow an applied path of answering relevant questions. For example, a few years ago a student asked me about how a withdrawal (W) on her transcript might hurt her chances of attending graduate school in psychology. I did not know the answer, but we designed a national study of psychology graduate admis – sions committee directors and looked at the impact of Ws on graduate admissions (to find out what happened, see Landrum, 2003). More recently, I had a student who applied to a regional conference to present some of our research that we had completed together. The conference rejected the paper (which happens) but provided no reason as to why the paper was rejected. Personally, I think student attendance at a conference should be about the learning experience, but not everyone agrees. So this student and I did a national study of psychology instructors to ask them about the role of student participation at conferences, as well as what cri – teria are important for student success at regional conferences. Most faculty see undergradu – ate participation at conferences as more of a learning experi – ence for the student than as the advancement of psychology as a discipline (see Haines & Lan – drum, 2008, for more details). The moral to this story—select a research topic that is relevant to you, and you may be able to answer a question of interest. You have an idea for your research paper that your instruc- tor approves of, and you have reviewed the literature, devel – oped your measures, collected After you have an idea for your research paper, have it approved by your instructor and start your research. What possible research topics are relevant to you? Creatas/Thinkstock lan66845_01_c01_p001-022.indd 17 4/20/12 2:43 PM CHAPTER 1 Chapter Summary data, analyzed the results—so now what? The major sections of your research paper (introduction, method, results, discussion) will tell (or retell) your story from start to fin – ish. See Chapter 2 for tips on how to prepare each of these sections. But remember that there is an important difference between designing and proposing a research project, and actually conducting that project. Part of the confusion comes from the word “research.” For instance, you are doing research when you are reviewing the literature and reading about past studies. But if you are collecting survey data, you are also doing research. The collection of data as part of a research project is usually accompanied by term “empirical.” Empirical research is the type of research where you are collecting new data from research participants. Applying Psychology Understanding tools like statistics and research methods can assist you in studying any topic in psychology that you want to. There is a classic saying that goes like this: Give a man a fish and he eats for a day; teach a man to fish and he eats for a lifetime. Certainly the content of psychology is important, but if you can learn, practice, utilize, and apply the methods of psychology, the journey ahead has endless possibilities. You might even be poised for a career as a psychologist or in a psychology-related profession—more about this in a later chapter. It is important to understand some of the basic core ideas of research designs. Students can hopefully become more confident in the skills and abilities they are developing, as well as gain content knowledge. If a student can learn to like (or at least appreciate) what research methods are all about, then that student has a good glimpse of what psychology is all about. If you are a practitioner of psychology (and not a researcher), you still need to be able to read and critically consume psychological literature. Chapter Summary P sychology is the science of human behavior, and psychologists rely on the processes and outcomes of the scientific method to inform and enlighten us about why we think, feel, and behave as we do. When we believe in the scientific method and follow its tenets, the conclusions we draw from science are powerful and informative. Scientific thinking is a specialized version of critical thinking that involves healthy skep – ticism, precise definitions, an appropriate reliance on evidence, serious self-reflection, and the acceptance of uncertainty. When psychological theories are formed following the research methods of science, our understanding of human behavior is advanced through the testing of specific hypotheses that can lead to insights about attit udes, opinions, and behaviors. Psychologists (as well as psychology majors) can apply the research methods of science to applied questions of interest, which become hypotheses to be researched and tested. Specialized databases allow researchers to review the available literature on a topic to help shape the questions to be answered by new research. The process of using research methods and a critical thinking approach provides us with valuable tools that can be applied not only to research questions of interest but also in the decision making that is required in everyday life. lan66845_01_c01_p001-022.indd 18 4/20/12 2:43 PM CHAPTER 1 Concept Check Concept Check 1. Which of the following is NOT an assumption of science, according to your text? A. Determinism B. Finite causation C. Empirical evidence D. Immediacy 2. Tilly wants to operationalize “anxiety.” To do this, she would A. remove anxiety from her research. B. consider anxiety to be the focus of her research. C. compare types of anxiety experienced in surgeries. D. define anxiety as it will be measured in her research. 3. It was determined that Clever Hans A. was well trained by his handler to count and do complex math problems. B. responded to subtle cues from humans about the correct answer. C. could read sophisticated hand signals from his handler and “cheated.” D. could add and subtract, but not do more advanced math problems. 4. A theory that accounts for previous studies’ findings and provides a theoretical framework is said to have A. subsumptive power. B. modifiability. C. internal consistency. D. parsimony. 5. The study conducted by the author regarding the effect of withdrawal grades on admission to graduate programs (Landrum, 2003) illustrated the idea that research may be A. related to the university experience. B. easily falsified through empirical evidence. C. an applied question of personal interest. D. conducted over a long period of time. Answers 1. D. Immediacy. The answer can be found in Section 1.1. 2. D. Define anxiety as it will be measured in her research. The answer can be found in Section 1.2. 3. B. Responded to subtle cues from humans about the correct answer. The answer can be found in Section 1.3. 4. A. Subsumptive power. The answer can be found in Section 1.4. 5. C. An applied question of personal interest. The answer can be found in Section 1.5. lan66845_01_c01_p001-022.indd 19 4/20/12 2:43 PM CHAPTER 1 Key Terms to Remember Questions for Critical Thinking 1. Which characteristics of critical thinking mentioned in this chapter do you think you would like to improve for yourself? Which of those characteristics do you see as current strengths, and which do you see as potential areas for improvement? 2. The story of Clever Hans is pertinent to hypothesis testing because a particular working hypothesis was believed until it was disproven. Do you think a work – ing hypothesis can be detrimental to the research/critical thinking process? Do you think it is better to go into a research situation with a completely open mind, and just look to see what might happen, or is it better to go into a research situ – ation with a working hypothesis and look for evidence that confirms or refutes your hypothesis? 3. Why is it so hard to tolerate uncertainty? By now you’ve taken a number of courses in psychology and in other subjects. Thinking broadly, what makes uncertainty so difficult to handle? Why do we need to know how the story turns out—can’t we just enjoy the journey without knowing the precise destination? Can you think of specific instances in your life where you tolerated high levels of uncertainty well, and can you think of a situation or two where you did not handle high levels of uncertainty so well? What were the key differences between those situations, and what can you learn from studying your own reactions to uncertainty? Key Terms to Remember anecdotal evidence Evidence based on a personal story or experience that is gener – ally not considered to be scientific or empir – ical support for a hypothesis, but could contribute to hypothesis development. causal relationship A direct relationship where an event occurs as a consequence of a previous event taking place. critical thinking A set of strategies designed to make an individual a better consumer of information through inquiry, interpretation, and problem solving. determinism The theory that all events are predictable and that if all the causes were known for an event, that event would be completely predictable. Also known as the “lawfulness of nature.” empirical evidence Evidence produced by science. fact The result of careful observation that offers a description of an event or behavior. fecundity The generation of new ideas; fruitfulness. See heuristic value. finite causation The concept that there are a limited number of causes for any effect or event, and that these causes are discov – erable and understandable. heuristic value A theory’s ability to motivate others to conduct research on the topic and generate new ideas about the world that we live in. See fecundity. hypothesis An educated guess that attempts to explain the facts that result from scientific studies. intersubjective testability When a theory generates hypotheses that are testable from an empirical standpoint. lan66845_01_c01_p001-022.indd 20 4/20/12 2:43 PM CHAPTER 1 Web Resources law A generalization for a collection of facts, but without explanation. Scientific laws are identified when no exceptions have been found to the law; scientific laws explain what has happened. nonsense syllables A consonant-vowel- consonant combination that is pronounce – able but has no inherent meaning, created by Ebbinghaus for memory research on himself. parsimony When a theory or idea is simple, yet complete. psychology The science of human behavior. savings Established by Ebbinghaus, the difference in time between when something is first learned and when it is re-learned. scientific method A method of studying the world around us by observing and developing theories through scientific hypotheses. skepticism The potential doubt that others may feel regarding the findings of scientific analysis. Scientists value this because they want evidence to either sup – port or refute a claim. statistical relationship A relationship between two variables that is found through the analysis of statistical measures. subsumptive power The ability of a the- ory to account for the results of prior stud- ies while offering a theoretical framework. theory An attempt to explain facts that are often tested as research hypotheses. values of science Outlined values that state that science places high value on theories that have the largest explanatory power; science values fecundity; science values open-mindedness; scientists require logical thinking in their explanations; science values skepticism; and science is self-correcting. Web Resources Access scholarly articles that a researcher may use for the purposes of developing his or her own independent research. This search engine allows for broad searches among numerous academic disciplines including, but not limited to, academic journals, gradu – ate theses,or professional publications. http://scholar.google.com/ This website provides access to a critical thinking community where the concept of criti – cal thinking is both defined and analyzed. It further goes on to explain the benefit of critical thinking technique and provides resources related to the topic. http://www.criticalthinking.org/aboutCT/define_critical_thinking.cfm This website outlines ways for psychological researchers to develop original, successful research ideas based on the paper by William J. McGuire. http://manyitems.tripod.com/miscellany/ideas.html Association of Psychological Science website. Use this resource to get involved with the association as well as be kept up to date on news, employment networks, and upcoming conferences and events. ht t p://w w w.psycholog icalscience.org/ lan66845_01_c01_p001-022.indd 21 4/20/12 2:43 PM CHAPTER 1 Web Resources American Psychological Association (APA) website. Use this resource to investigate top – ics in psychology, as well as be involved with the APA and be kept up to date on current research and topics in the field. http://www.apa.org/ lan66845_01_c01_p001-022.indd 22 4/20/12 2:43 PM 2 Practical Matters for Psychology Projects Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • appreciate the role that American Psychological Association style and format play in the preparation of research papers in psychology. • prepare a manuscript in 6th edition AP A format. • understand the overarching principles that govern research with humans and animals, including beneficence, respect for persons, and justice. • comprehend the importance of the protection of human participants and the role of the Insti – tutional Review Board in providing those protections. • differentiate between anonymity and confidentiality in human subjects research and under – stand the role and importance of debriefing in psychological research. Science and Society/Superstock lan66845_02_c02_p023-062.indd 23 4/20/12 2:44 PM CHAPTER 2 Introduction Introduction A s a developing scholar in psychology, you need to be able to critically digest journal articles, book chapters, and other information and evaluate this information from multiple perspectives using an interdisciplinary approach. Sometimes beginning students take a shortcut when reading journal articles and only read the Abstract, skip – ping the rest. As a student of psychology, you’ll want to read the entire journal article so that you can eventually write your own scientific story describing your applied project. The sections of a journal article provide an essential road map that shows how scientists think and why we design research the way we do. If you want to be a communicator of social science research results, you need to begin to master writing in American Psychological Association (APA) style and format. Writing in APA style means that the writing attempts to communicate objectivity, credibility, and an evidence-based approach. Not only does APA publish its Publication Manual (APA, 2010) to aid in format and style issues, but there are a number of other helpful writing guides available to you. Good science relies on public communication, and public communica – tion relies on good writing skills. When all psychologists follow a similar format for the publication of research results, consistent presentation of materials adds to the clarity of the presentation as well as efficiency. At the turn of the twentieth century, publications in psychology shifted away from Euro – pean dominance to more prevalence in America. In 1928, a 6-member panel attending a conference of editors and business managers for anthropological and psychological peri – odicals (truly an interdisciplinary effort) published a 7-page report in the 1929 volume of Psychological Bulletin titled “Instructions in Regard to Preparation of Manuscript.” This guide was revised on multiple occasions, leading to the sixth edition of the Publication Manual to be published by APA in 2010. Odds are, you will not be instantly good at writing in APA format, but you must practice. Throughout this chapter we’ll discuss helpful tips to practice and gain confidence in your writing skills. One of the basic tenets of science and scientific knowledge is com- municability (one other basic tenet of scientific knowledge is replication). APA format facili – tates communication of scien – tific, psychological knowledge by the reporting of results in a consistent and predictable for – mat. Any paper written in APA format has information pre – sented in the following order: title page, abstract, introduction, method, results, discussion, and references. Knowing the parts of the manuscript and where they are located gives an advantage Writing is a way to help communicate research findings, and writing in APA format is an important part of writing a professional research paper. iStockphoto/Thinkstock lan66845_02_c02_p023-062.indd 24 4/20/12 2:44 PM CHAPTER 2 Introduction Voices from the Workplace Your name: Kimberly C. Your age: 43 Your gender: Female Your primary job title: Director of Marketing & Public Relations Your current employer: ResCare,Inc. How long have you been employed in your present position? 4 years What year did you graduate with your bachelor’s degree in psychology? 1986 Describe your major job duties and responsibilities. I am responsible for marketing the wide array of services my company offers, in coordination with regional Administrators and staff. I develop brochures, press releases, newspaper articles and other advertising. I also sit on the Senior Management Team of my company, allowing me to participate in long term strategic planning and oversight of my company, which employs over 600 people. I am par- ticularly involved in business development, which includes researching and involvement in business startup. What elements of your undergraduate training in psychology do you use in your work? Currently, I use my understanding of psychology to guide employees in our mission, vision and core values. Relationship issues are valuable in the workplace, and I believe my training in psychology has helped in that area. It helps me to relate to other staff members, including psychiatrists, therapists and direct support staff in problem-solving and looking for growth opportunities. Additionally, my degree in psychology opened the door for me to get into the field of providing services to persons with seri- ous and persistent mental illness and other disabilities. My experience coupled with my education has allowed me to be successful and therefore promotable. What do you like most about your job? Advocating for persons with disabilities to empower themselves, by influencing the culture of my company. What do you like least about your job? Government bureaucracy. (continued) to the reader; you may not understand the jargon used, but you know there is a descrip – tion of how the study was conducted in the Method section, and the statistical findings of the study are recorded in the Results section. For some papers, you might not have all of these sections, such as an Abstract or Tables. Other organizations adopt other formats, such as Turabian, the Chicago Manual of Style , and Modern Language Association. APA is not necessarily superior to any of these other writing styles and formats, but APA format is the style for most writing in psychology. By mastering APA format you will demonstrate to your teachers that you are a serious stu – dent and that you have the ability to follow instructions and pay attention to detail—two characteristics that are important in college and beyond. lan66845_02_c02_p023-062.indd 25 4/20/12 2:44 PM CHAPTER 2 Introduction Beyond your bachelor’s degree, what additional education and/or specialized training have you received? A few MBA classes; Sales and marketing training when I left the human services field for 6 years to go into the financial services field; countless internal and external training sessions on service provision, leadership, marketing, business administration. What is the compensation package for an entry-level position in your occupation? A person with a bachelor’s degree in psychology could begin at the $28,000–$30,000 per year level. However, direct support staff typically start out at $8–$10 per hour. What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? Health Insurance, Life Insurance, Dental Insurance, 401(k) with nice match, PTO, Holidays off; and through my employee-owned company I have stock ownership. As a stockholder, I have long term care insurance for myself and my spouse with a partial company premium payment. What are the key skills necessary for you to succeed in your career? Ability to relate well with people, be a reliable team player and leader, Have a positive “can-do” attitude. Technical skills include computer skills, writing skills, public relations, and strategic planning skills. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? General Psychology, Abnormal Psychology, History of Psychology (different schools of thought), Speech, Literature and Creative Writing, Contemporary Social Problems, Sociology, Behavioral Psychology. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? I belonged to the Student Senate and Alpha Phi Omega (a service co-ed fraternity), but I can’t say that either were keys to my success. As an undergraduate, do you wish you had done anything differently? If so, what? I wish I had taken more business related classes. What advice would you give to someone who was thinking about entering the field you are in? It’s a wonderful and rewarding career. I left it for a period of time, and have never regretted coming back. However, don’t expect to walk into a high paying, professional position out of college. Most peo- ple will need to get experience working in the field. Opportunities for growth and advancement are available. Hard work and good work ethics will take you far. This is also a good field for those looking for flexibility with an ability to make a difference. If you were choosing a career and occupation all over again, what (if anything) would you do differently? I probably wouldn’t do anything differently. My initial plan was to go straight to graduate school to get my master’s in Psychology after I graduated with my bachelor’s. However, that did not work out and I am glad. I am better suited for a career in Health Care Administration and Business Management than as a therapist or counselor. I still hope to complete my MBA in the future, but I’m glad I have work experience instead of going immediately to graduate school. Copyright . 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. Voices from the Workplace (continued) lan66845_02_c02_p023-062.indd 26 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer 2.1 Writing Scientifically: A Brief Primer A lthough it may be hard to believe, the format and outline of an APA research paper provides the structure to tell a scientific story. The sections of that story, with proper labels and brief descriptors, are presented here. Major Sections of an APA Manuscript Title page (Take credit)• Author’s name, affiliation • Other information as your professor requests • Page numbering (header) and running head information Abstract (Quick summary) • No more than 120 words • Some assignments will not require an abstract Introduction (What you are studying) • Introduce the problem • Develop the background • State the purpose and rationale for the present study Method (What you did) • Participants, Materials, Procedure • Should be in enough detail to replicate if desired Results (What happened) • Presentation of statistical outcomes; tables and/or figures if necessary • Presentation, not interpretation Discussion (What it means) • Was there support for the research idea? Did the study help resolve the original problem? • What conclusions can be drawn? Suggest improvements, avenues for further/new research Reference section (Give credit where credit is due) • Starts on its own page • Authors listed alphabetically by last name, no first names used, only initials • Be sure all citations in the text are referenced • Shows your scholarly ability and how you did your homework The Major Sections of an APA Research Paper Of all the components of an APA research paper, there are four sections that stand out: the Introduction, Method, Results, and Discussion. Introduction Although the Introduction section is the introduction to your paper, it won’t appear until page 3 of your APA style research paper (the title/cover page is page 1, and the Abstract lan66845_02_c02_p023-062.indd 27 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer is presented on page 2). The introduction is truly a key section of the paper because it lays the foundation for everything that is about to happen in your research. Think of it as the backstory of a movie plot; you are providing your reader with a context for everything that is about to occur. Your introduction will have three major goals: (a) introduce the research problem, (b) develop the background, and (c) state the purpose and rationale for the research paper, including hypotheses. For some paper assignments, all you may be asked to do is this introduction section; this might be analogous to what some call a term paper. As you become more proficient in your writing, you might not need this template anymore, or your might omit items, and you might even violate a rule or two. But as you’re starting out, try following these five steps, in this order. 1. In the first paragraph of your research paper, you want to convince the reader that your issue is an important one, and worthy of study. You can be convincing a number of ways. You might cite some statistics about the behavior or phenom – enon you are studying, to convince the reader that this issue has widespread effects. Or, the behavior you are interested in is pervasive in our daily life, and matters to nearly everyone. The first paragraph of your Introduction should grab the reader ’s attention. If you can’t make your topic sound important, is it? Dunn (2011) offered advice for possible openers in the introduction, which he called opening gambits—see the following for examples. Possible Openers for an Introduction Section An everyday experience that readers will immediately recognize—”Studying is an essential component of a student’s life that appears to be related to success in college and beyond.” The absence of research in an area important to understand—”Even though self-confidence is an important marker for whether students continue to study or not, the metacognitive literature about how students develop self-confidence is wanting.” A rhetorical question that redirects readers to examine their own feelings about the issue—”How would you feel if you invested (and your parents, and fellow taxpayers) four or more years into a college education, and that education yielded very little measurable impact on the development of higher-order skills and abilities?” A compelling fact or statistic that is typically surprising—”When tested for change over time on a comprehensive measure of critical thinking from their sophomore to senior years, only 27% of college students experienced a significant improvement in critical thinking skills.” A metaphor or an analogy that joins two seemingly disparate beliefs or ideas—”The process used to assess successful acquisition of higher-order skills in college seems akin to throwing spaghetti against a wall to see if it is done—assessment appears to identify some students who “stick” and others who do not.” A historical reference that helps to indicate change over time—”The perceived decreasing performance in college students today may be attributed to a growing number of distractions—for instance, in 1972 the average student studied 23 hours per week and worked 14 hours per week. By 2009, students worked an average of 36 hours per week while reporting just under 10 hours of studying per week.” lan66845_02_c02_p023-062.indd 28 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer 2. In the next section of your Introduction you start to lay that foundation—the context for the research that has come before your research project. You should be reading a variety of journal articles in psychology, which may be useful for the type of APA-style paper that you are writing. The reader can “tell” when you get to this part of the introduction, because citations appear in the text (more on in-text citations later in this chapter). Here is where you demonstrate some of your scholarly abilities; you provide the reader with the intellectual path you took to shape and form your ideas. The citations are critical because they show off your academic pedigree—writing like a psychologist means that you write with accurate attributions and properly use APA format to do so. This part of your introduction might span more than one paragraph. If your topic is particu – larly complex or has a long history, it might take two or three paragraphs (or more) to present the relevant work and shape your story. Essentially, you are answering the question “What is already known?” (Dunn, 2011). 3. You started big and broad with your opening gambit, and then filled in some of the backstory for the reader, providing a solid context for what research has occurred before yours. In this next section of the Introduction you make a statement of the problem to be solved. That is, your review of the literature has revealed a gap or hole in the field, and in this section you address that gap or hole directly. There is an unsolved problem or issue that needs to be solved. In this section you identify what part of our knowledge about a behavior or group or cultural phenomenon is incomplete. This is your statement of purpose that answers the question “Why is this study being conducted?” (Dunn, 2011). You could start the sentence like this: “The purpose of this study is to. . . .” 4. After you have set the stage, you provide a glimpse of what is about to hap – pen—call it “coming attractions,” if you will. You provide a brief snapshot of what is about to happen, in a sentence or two. You can allude to the participant population to be tested, how this study fills the gap, or a preview of the meth – odology used in the study. Here is the place where you can define and describe key variables. 5. You complete your introduction by being as specific as you can about the hypotheses you intend to test. Make your predictions here. The more careful and precise you are in this section, the easier later sections will be. For example, if you make some predictions about gender differences here, this will help you determine the appropriate statistic test when comparing two groups later in your Results section. Being detailed here will pay off later. For each hypothesis, start the sentence like this: “I hypothesize that . . .” Yes, first person voice is acceptable in APA format. The Introduction section to a research article follows what Bem (1987) refers to as the hourglass shape. Your introduction starts big and broad (at the top of the hourglass), and then becomes more and more specific (as the hourglass narrows). This is an excellent anal – ogy to follow, and if you remember it along with the idea that you are telling a story, you’ll be on your way to writing like psychologists write. lan66845_02_c02_p023-062.indd 29 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Method Immediately following the Introduction section is the Method section. The Method sec – tion provides the nuts and bolts of how your research was conducted. The subsections of the Method section (participants, materials, procedure) provide the blueprint for some – one who wants to replicate (i.e., repeat) your study. A key fea – ture about the Method section is that we give away the details— we tell others how the study was conducted and we don’t keep secrets. The ability to repeat a study (replication ) is an important part of the research process. In case something odd or strange happened with the first study, the ability to replicate a study means that we could confirm or deny the original findings. Replication is important when unexpected results are discov – ered and additional confirma – tion is necessary. Replicated studies help build the case for both validity and reliability of the results. So the Method section, although not very glam – orous at first blush, is vital to our advancement of understanding various areas within psychology. Your Method section has three subsections as follows. 1. Participants . In this subsection, you describe the characteristics of the individu – als who completed your study (if you were studying animals, you’d describe them here too). For humans, you would usually report the number who partici – pated, the average age (a measure of variability should accompany any average reported), perhaps a breakdown of the number (or percentage) of males and females, and other relevant demographic characteristics. Describe briefly how your participants were recruited and how they were rewarded for their partici – pation, if at all. How did you select participants, or did participants self-select or volunteer to complete your study? If participants dropped out of your study for any particular reason, describe those reasons in this section. Describe as much as you can about how the sample was obtained. 2. Materials . In this subsection you describe the “stuff” that you used to make your research happen. The Materials subsection provides the details of the actual items or objects that were used to carry out the study. For example, if you asked survey questions, then in this section you would describe those survey items, either how you created original items or giving credit to those who created the survey items. Did you pilot test that original survey prior to administration? Ideally, the sur – vey items you chose would already have validity and reliability statistics avail – able, and you would cite those in the Materials subsection. If you were doing a study by presenting stimuli on a computer, you would describe what was being The Method section in a research paper provides information about the participants, materials, and procedures to help others replicate your study. iStockphoto/Thinkstock lan66845_02_c02_p023-062.indd 30 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer displayed on the screen, for how long, and the types of responses that partici – pants were being asked to provide (“Participants were tested in a computer lab using PC/Windows operating system”). If the study included highly technical equipment, or equipment that is not widely available, you might have an Appa- ratus section rather than (or in addition to) a Materials section. In the Apparatus section, you would provide enough details so that others could construct what you constructed, or re-create your apparatus to such an extent that the study could be replicated. An Apparatus section is not common in Method sections cur – rently, but you may need one if you are creating or modifying an existing piece of equipment for a new purpose (for example, the creation process used to modify an existing psychology application [app] for an iPad). 3. Procedure . In this subsection, you “walk” the reader through the process that you used to conduct your research, step-by-step in chronological order. Were partici – pants tested individually or in groups? If they completed a survey, was it online, bubblesheet, or paper and pencil? If you conducted a true experiment, describe how the experimental group and the control group were randomly assigned. Describe how participants were assigned to groups or subgroups, addressing issues such as randomization or counterbalancing (as appropriate). Were the instructions critical to the design of the study? If so, be sure to include them in this section. Describe the debriefing process used at the end of the study session. Again, the overarching goal of the Method section is to provide enough instruction so that if someone wanted to replicate your study, they could. As you think about research and data collection, make sure you read and study the Method section of others so that you can better understand how research is conducted from start to finish. Results Directly following the Method section is the Results section. At this point we’re still in the narrow crook of the hourglass—that is, the Results are very specific to your study (not big and broad and general, like the Discussion). The Results section tells the reader what happened—the outcomes, typically from a quantitative or qualitative viewpoint. Start your Results section with the outcome that is most impor – tant to your research (Salovey, 2000). You’ll want to be sure that you use proper APA format to describe the outcomes, and be sure to italicize items like t or F when reporting inferential statistics, and the p in p value. In fact, the preference in the new Publication Manual (2010) is to report exact p values whenever possible, and to report the effect size with inferential statistics. With qualitative data, you’ll want to fully explain your anal – ysis and extraction methods, The results section is where you describe how you obtained and analyzed your data. iStockphoto/Thinkstock lan66845_02_c02_p023-062.indd 31 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer providing the path for how you came to your outcomes. Toward the end of your Results section, summarize for the reader what has happened. In some quantitative cases you may be reporting an effect size along with your statistical analysis, to help the reader understand if the statistically significant outcome has any practical significance. With a very large N, relatively small correlations, for example, can be statistically significant, but they don’t mean much—this is where effect size statistics can help. In the Results section, it’s all about presentation (the factual outcomes, if you will), not interpretation. Bem (2004) offered a helpful step-by-step guide for the thorough presentation of informa – tion in the Results section: • Verify that your study was successful in setting up the conditions needed to adequately test your hypothesis (nothing major went wrong in the conduct of the study). • Describe your overall approach to data analysis, including the methodology used to obtain your dependent variable measurements. • Provide a brief reminder of the main conceptual question/hypothesis, and a reminder about the basic tests performed and behaviors measured. Why remind – ers? Because sometimes readers will read parts of a research paper (or journal article) out of order. The reminders are a courtesy to save readers from the extra work of looking for the context to interpret the section they are reading. • Answer your hypotheses as clearly and unequivocally as you can, using words, then in numbers, using APA format. • After addressing the major hypotheses of the study, address other findings and/or surprises that emerged. Use the same format—describe what happened clearly, in words, followed by your statistical evidence in numbers. • You may want to organize your Results section into logical subsections if that will help the reader follow the story. Be sure to use the proper APA style head – ings as signposts, just as you did with the Participants-Materials-Procedure subsections of the Method section. • As you move from paragraph to paragraph in the Results section, try to provide smooth transitions between paragraphs, emphasizing the logical flow of your hypothesis testing and the outcomes of your research. Plonsky (2006) provided additional advice on what not to do in a Results section: • Do not discuss the implication of the results in the Results section; that is saved for the Discussion section. • Do not discuss the alpha level or the null hypothesis, because most readers in the scientific community will already understand these assumptions. • Do not organize subsections of your Results section by type of analysis (all the t tests in one paragraph, all the correlations in the next); organize subsections by variable to be studied or hypothesis to be tested. • Do not present the raw data collected, unless that is part of your instructor ’s assignment; and do not use the word “proved,” because in science we never prove anything; we only disprove competing theories and hypotheses until one logical explanation is left standing—and hopefully, it is our working hypothesis. This is an example of the hedging language used in psychology (Dunn & Smith, 2008)—scientists do not “prove” anything. lan66845_02_c02_p023-062.indd 32 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Discussion The Discussion section is about interpretations and conclusions. First, you’ll draw a conclusion about whether or not your null hypotheses were rejected. More generally, however, you work to determine (based on your evidence) if the data are supportive of your working hypothesis. Then you place your study in the context of the studies from the past—did you fill that knowledge gap you were attempting to fill? You place your results in the context of those studies. Did you confirm previous findings, or did you discover something new? What conclusions can be drawn from your research findings? Do these conclusions have broader, policy implications—that is, can you generalize beyond your study? Then discuss the limita – tions of your research, but only briefly. No study is ever perfect, so all studies have limitations. You conclude your Discussion with the “take home” mes – sage—what is the central mes – sage that you want the reader to come away with? That is, if the reader could remember only one idea from your study, what would you want that idea to be? When you start writing your Discussion section, it may be helpful to have a template that guides you in this process. The template helps you to learn the style and format as you begin to write a Discussion section. These aren’t rules per se, but they’ll help you get started. • Lead off your Discussion section with the main finding of your research—your take-home message. If your reader were going to remember only one piece of information from your entire study, what should it be? Whatever it is, it should be presented in the first paragraph of your Discussion section. • Next, you address each of your hypotheses that concluded your Introduction. In the Results section you presented the numerical/statistical results or qualitative outcomes, but in the Discussion section you interpret these results. Make sure that somewhere in the Discussion (usually this section of the Discussion) each hypothesis is addressed—either supported or refuted. • Now put your research in the context of the studies from the past that you reviewed in your Introduction section. Remember back in the Introduction sec – tion where you laid the foundation for your study, but you described some gaps or holes to be filled in the literature? Now, fill the gaps. Did your study help address a question from the literature? Perhaps you answered one question but raised three other questions (that’s OK; it happens frequently). Share with the reader the new questions that you think should be addressed in future research. The reader will know when he or she reaches this part of your Discussion because citations from the Introduction will reappear. When writing your discussion section, include why you rejected or accepted your null hypotheses. Also have a central message that you want the reader to take home from your paper. Hemera/Thinkstock lan66845_02_c02_p023-062.indd 33 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer • The perspective in the paper continues to become broader as we approach the bottom of the hourglass. In this section of the Discussion you can speculate about what your results mean, but make sure your conclusions are properly labeled as speculation. In other words, what are the broader implications of your study? Would the results have any impact on public policy? Might your outcomes sug – gest different advice to give? Here you generalize about your research—what might be the practical applications of the results of your study? • Next, briefly present some of the limitations of your study. Be fair, but don’t beat up on your study. Perhaps the sample wasn’t very random or other difficulties occurred during your study. Remember that no study is ever perfect. Briefly point out your study’s shortcomings and move on to future recommendations. If you were giving advice to a researcher who was doing the next study in this area, what would your advice be? • Finally, end your Discussion with a brief paragraph that summarizes major points made earlier. Essentially, you remind the reader of (a) the take-home mes – sage, (b) the importance of your study in adding to the literature, and (c) the gen – eral importance of the topic itself. This helps justify to the reader the worthiness of your work, and it provides a broad completion to the bottom of the hourglass. As Bem (2004) put it, “End with a bang, not a whimper” (p. 203). Attention to Detail: Title Page, Abstract Tables, Figures This chapter presents the formal order in which sections of an APA-style paper appear; however, you should note that you might not always have every section in every paper you write. For instance, if you were writing a literature review paper, you might not have an Abstract , nor tables or figures necessarily. One other detail to mention here is that even though this chapter follows the order of how the sections of the paper are assem – bled, this is typically not the order you would follow when writing your paper. The title page (also called the cover page) and Abstract are typically the last sections to be written, yet they are the first two pages of an APA formatted paper. A properly formatted APA paper appears plain, but there are plenty of details to attend to. You’ll use a Times New Roman 12 pt. font, double spacing, and 1-inch margins on all four sides of the paper. There are precise rules for the preparation of the manuscript. For example, on the cover page, you’ll have the running head and the page number at the top of every page of your manuscript, all inside the top 1-inch margin. (A running head is a short description , such as ACADEMIC STRESS AND NUTRITION, that appears across the top of each page or every other page of a journal.) The running head label is in mixed case, but the actual running head is all caps and limited to 50 characters (the words “Running head” only appear on page 1). Finishing off your title page comes the title of your article (12–15 words is the maximum recommended length of your title), your name, and then your affiliation (the school you attend). This is all double-spaced and centered toward the middle of the cover page. If you have trouble coming up with a title for your research paper, the generic version of a title is this: “The Effects of the Independent Variable on the Dependent Variable”—just substitute your independent and dependent variables into the generic title. Other typi – cal forms of the title may start with “A Study of . . . ,” “An Investigation of . . . ,” or “An lan66845_02_c02_p023-062.indd 34 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Experiment on. . . .” (Sternberg, 2000). The title you select is more important than you might think, especially if you continue to write in psychology as part of your career. The title is important in capturing attention, its indexing in databases, and helping to form first impressions (Sternberg, 2000). Sternberg provided a fine example of how a title can make a positive first impression. A seminal article in cognitive psychology and human memory research is George Miller ’s (1956) article titled “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information.” Imagine if that article had been titled “Limitations on Information-Processing Capacity: A Review of the Literature” (Sternberg, 2000, p. 38). The Publication Manual (2010) also indicated that Author Notes should appear toward the bottom of the cover page—you should consult with your instructor to see if Author Notes are necessary. Depending on the type of paper you are writing, your instructor might not want a title page at all. Instead of the affiliation your instructor might want you to include the course number and name. Although we’re discussing standard APA style instructions, always heed your instructor ’s directions, because that is who is grading your work. The Abstract follows the title page. This might be the most difficult single paragraph you will write, and it is important. The Abstract provides a quick synopsis of what hap – pened—a summary of the outcomes. APA style limits your Abstract to 120 words, so you must be succinct (the word count feature in Word can easily calculate this value for you). Try writing your Abstract last, because you can’t summarize everything that has happened until it has happened. Your abstract should cover the following areas (prefer – ably in this order): (a) background and purpose—about one sentence, (b) hypothesis (or hypotheses)—one sentence, (c) sample tested—one sentence, (d) design utilized—one sentence, (e) method and measures used—about two sentences, (f) results—about one or two sentences, and (g) conclusions and implications—one or two sentences (Dunn, 2011). None of the items above could be more than two sentences and still stay within the 120 word (maximum limit). The abstract paragraph, appearing on page 2, is never indented. Also, in the Abstract, when referring to numbers, use the actual numerals (5, 7) rather than spelling them out (five, seven). In the Abstract, the numerals under 10 rule does not apply. After completing your Abstract, add keywords (no more than five) that would help others if they were searching for your manuscript in a database. If you continue writing in psychology (and start publishing in psychology), Abstracts become vital because they are indexed and cataloged into databases like PsycINFO. When we identify articles to retrieve and read in psychology, this identification is largely based on the Abstract. One last “attention to detail” topic regarding the manuscript covers tables, figures, and an appendix. Tables are typically used to present quantitative materials, and figures are typi – cally used to share graphical or pictorial results. An appendix is not often used, but when it is, it acts as a depository for information that was important for the preparation of the research, but not so important that it had to be included in the text or flow of the research report. In general, you’ll want to use these sparingly. Although it may be odd to think this way, you should only use these options a storytelling device—perhaps analogous to determining if a story should contain illustrations or not. That is, only use a table or figure if it helps you tell a better story. In general, it’s easier to write in text in a paragraph than to prepare a table or figure in APA format. But if you have a great deal of quantitative data, a well-prepared table can be efficient and help to tell your story. A well-placed bar graph lan66845_02_c02_p023-062.indd 35 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer showing a significant interac – tion can also be an important figure to help express the out – comes of your study. There are specific sections of the Publica- tion Manual that will help you in the preparation of tables and fig- ures. In fact, this can be so com – plicated at times that there are specific APA reference guides to help you with tables (Nicol & Pexman, 2010a) and fig – ures (Nicol & Pexman, 2010b). Always be sure to check with your instructor about tables, figures, or an appendix before going to the trouble of prepar – ing them according to APA style. Dissecting the Journal Article Reference Your References section is vitally important to your research paper. The References sec – tion provides the “intellectual path” that you followed to form and write your paper. The task of preparing citations and references “. . . is, however, one of the most important topics regarding manuscript preparation because through citations and references you make or break your reputation as a careful and thorough scholar” (Smith, 2000, p. 146). To this end, many academics prefer that you not cite sources like Wikipedia. Although these online resources may help you explore possible research ideas, because of the lack of systematic peer review, such easily editable online resources sometimes lack credibility and veracity. The References section is not a bibliography—you will not be listing every resource that you explored. The References section will contain a listing of every citation that you used in the paper—and every reference you used in the paper could be presented as a citation in your References section. Preparing a References section properly may be the ultimate task that shows off your attention to detail. Note that your References starts at the top of its own page, immediately following the Discussion section; you can insert a page break to force the text to the top of the next page. Another word processing tip: In Microsoft Word, type in the reference in perfect APA format, highlight it with your mouse, and hit control-T. The hanging indent function will be performed—no need to use tabs or manu – ally space the lines over. Given the complexity of preparing different reference types in APA format, you may want to use software programs (e.g., EndNote Plus™, WPCitation™, Manuscript Man – ager™) that aid in the bibliographic gathering of reference information. Some pro – grams aid in the preparation of research papers or manuscripts directly. While these types of programs are fine for helping you track and organize bibliographic citations, I would caution you not to use them in manuscript preparation. Why? First, if you let the Charts and graphs can help convey the results of your study, especially if you have a large amount of quantitative data. iStockphoto/Thinkstock lan66845_02_c02_p023-062.indd 36 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer computer program do the APA formatting of your references in text and in the References sec – tion, then you won’t learn the details yourself (using a famil – iar analogy, we teach children to do math by hand prior to giv – ing them a calculator). Second, if your instructor makes devia – tions from APA style (such as two spaces after a period rather than one), odds are you can – not alter the software to follow some APA rules and not others. I recommend that you conquer APA format on your own first, and then if you wish, rely on a computer program to ease the workload. The components in Figure 2.1 should be very helpful in identifying parts of a reference citation. Read on for more examples of typical reference citation formats. Note that in an APA-formatted paper, the references would be double-spaced. Author’s last name leads the reference Gurung, R. A. R. (2005). How do students really study (and does it matter)? Teaching of Psychology, 32, 367–372. The year the item was published, in parentheses, followed by a period Note the capitalization of the title—different than you might expect Journal name italicized Journal reference ends with a period Page numbers of journal article, inclusive Volume number italicized, but no issue number included Hanging indent used (Ctrl-T) when reference continues on second line—double spaced Only use initials for the first and middle names With annotations, this graphic labels the parts of a journal article citation presented in APA format. In a research paper, the only difference would be that this reference would be double spaced in the References list at the back of the paper. Figure 2.1: Anatomy of an APA journal article citation The References section is an important part of a research paper because it provides the sources of information you used as citations in your paper. Hemera/Thinkstock lan66845_02_c02_p023-062.indd 37 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Reference Format Examples Journal article with no digital object identifier (DOI) assigned Gurung, R. A. R. (2005). How do students really study (and does it matter)? Teaching of Psychology, 32, 367–372. Journal article with DOI assigned Kazdin, A. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychol – ogist, 63 , 146–159. doi:10.1037/0003-066X.63.3.146 Book Wilson, J. H. (2005). Essential statistics. Upper Saddle River, NJ: Prentice Hall. Chapter in an edited book Crawford, M. P. (1992) Rapid growth and change at the American Psychological Association: 1945 to 1970. In R. B. Evans, V. S. Sexton, & T. C. Cadwallader (Eds.), The American Psychological Associa- tion: A historical perspective (pp. 177-232). Washington, DC: American Psychological Association. General Internet citation Burgess, C. (2000). Psychology graduate applicant’s portal. Retrieved from http://www.psychgrad.org Magazine article Roig, M. (2008, Winter). Avoiding those little inadvertent lies when writing papers. Eye on Psi Chi, 12(2), 31–33. Lecture notes Ferguson, T. (2004). Chapter 12: Social development [web site]. Retrieved from Utah State University web site http://www.usu.edu/psycho101/lectures/chp12socdev/socdev.htm Conference poster presentation Estow, S., Lawrence, E. M., & Adams, K. (2007, October). Practice makes perfect: Improving plagiarism identification in psychology majors . Poster presented at Beginnings & Endings: Best Practices for Introducing and Bringing Closure to the Undergraduate Psychology Major conference, Atlanta, GA. Conference paper presentation Appleby, D. (1999, April). Advice and strategies for job-seeking psychology majors . Paper presented at the meeting of the Midwestern Psychological Association, Chicago, IL. Audio podcast Fogarty, M. (Host). (2008, September 19). Complex-compound sentences [Episode 136]. Grammar Girl Quick & Dirty Tips for Better Grammar. Podcast retrieved from http://grammar.quickanddirtytips.com Newspaper article available online Allen, A. W. (2008, September 19). ‘Clickers’ let teachers see who’s really learning a lesson. Idaho Statesman. Retrieved http://www.idahostatesman.com/eyepiece/v-print/story/507544.html Journal article on an Internet-only journal Taylor, J. G. (1998). Constructing the relational mind. Psyche, Volume 4. Retrieved from http://psyche. cs.monash.edu.au/v4/psyche-4-10-taylor.html Chapter or section of an Internet document National Center for Education Statistics. (2008). Postsecondary education. In Digest of educational sta- tistics 2007 (Chap. 3). Retrieved from http:// http://nces.ed.gov/programs/digest/d07/ch_3.asp lan66845_02_c02_p023-062.indd 38 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer One question you may be asking is “What’s a doi?” A doi is a digital object identifier, and it is code that is now used on some resources being published into the literature, primar – ily journal articles. The doi code provides a unique numerical identifier of the permanent location of the electronic file on the Internet. For example, Alan Kazdin published an arti – cle in 2008 in the American Psychologist about evidence-based treatment and practice. That article was assigned a doi, which is doi:10.1037/0003-066x.63.3.146 The doi is useful in a couple of ways. First, you could go to the website http://www.cross- ref.org and enter the doi in the DOI Resolver box, and it should take you to the article. Or, you can add this—http://dx.doi.org/—directly in front of the doi and then paste into your web browser, like this http://dx.doi.org/doi:10.1037/0003-066x.63.3.146 What’s handy about the new doi system is that it provides a permanent home on the Internet for electronic files. You will still need to gain access, perhaps through your library online access, or through a service that you subscribe to. But the doi, if the article has it, should help you find it more easily (and more consistently) on the Internet. Managing Citations Good scientific writing places ideas about variables, groups, cultures, or historical periods in context. That is, part of the story is the backstory that contributes to our current state of knowledge about a phenomenon, and in a research paper, filling the gap or hole in the knowledge is the goal. To accomplish this you must be familiar with the existing literature, which is why you conduct your library research on your topic, you extract materials using the Notecard method, and you synthesize those materials looking for common themes or threads by arranging your idea notecards into coherent paragraphs. An essential compo – nent of this task is the ability to cite the work of others in your own work. As a psychology student, you are already becoming accustomed to this practice by now. Remember reading your introductory psychology textbook? You would be reading a par – ticular paragraph, but the flowing text would be interrupted by last names and a year, sometimes in parentheses, and sometimes not in parentheses. This practice, called citing a reference, is vital to scientific writing. We cannot borrow others’ ideas without proper attri – bution. The ability to cite (and properly cite using APA format) is one method where you show you are developing into a scholar. Students sometimes worry that an introduction/ literature review is not very original, because it is so filled with the citations of others’ work. However, the originality comes from the method by which you put those ideas together— your unique contribution is the thread or synthesis or common theme you identified and then documented with your citations. The ability to identify common themes where they exist is a highly sought-after intellectual skill; therefore, using proper citation methods (and reference lists) helps you demonstrate your developing abilities as a scientist. Although there can be many variations on a theme, you’ll basically choose from three ways to present citations in the text of your paper: (a) author name(s) and publication year outside of parentheses; (b) author name(s) outside of parentheses, publication year lan66845_02_c02_p023-062.indd 39 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer in parentheses; and (c) author name(s) and publication year inside parentheses. How do you decide which format to use? It might be wise to consider the overall flow of the para – graph and to make a selection that avoids the passive voice. See the Table 2.1 for sample sentences using the variety of forms. Table 2.1: Examples of citation styles with varying numbers of authors Author and publication year outside parenthesesAuthor outside parentheses, publication year inside parentheses Both author and publication year inside parentheses One citation, one author only In 2004 Bem published a comprehensive article addressing critical issues in writing the empirical journal article. In a recent book, Bem (2004) offered key suggestions for writing the empirical journal article. Well-organized and pertinent advice about writing the empirical journal article already exists (Bem, 2004). One citation, two authors It was in 2006 that Calderon and Austin offered cogent suggestions for writing in APA style.In an excellent chapter by Calderon and Austin (2006), cogent examples of proper use of APA format are presented. The ability to write clearly in APA style is a marketable skill for students (Calderon & Austin, 2006). One citation, three authors, first time cited Forensic researchers Dewey, Cheatem, and Howe noted in 2010 that admissions to graduate programs may be surging due to the popularity of certain TV shows.Dewey, Cheatem, and Howe (2010) suggested that television shows may be publicizing the forensic psychology specialization to a wider audience than previously thought. The number of students applying to forensic psychology programs greatly exceeds the current capacity (Dewey, Cheatem, & Howe, 2010). One citation, three authors, subsequent citations In 2010, the question of interest asked by Dewey et al. was about the role of television in promoting forensic psychology. The outcomes suggested by Dewey et al. (2010) indicate that television shows contain inaccurate portrayals of the day- to-day activities of a forensic psychologist. Current data lead to the conclusion that applications will soon double the number of graduate school positions available in forensic psychology (Dewey et al., 2010). There may be occasions where you may need to reference two or more articles in the same sentence. This is usually accomplished inside of parentheses. You separate the references with semicolons, and the order of presentation is by first author ’s last name, not year of publication. An example would look like: Excellent advice for preparing research papers in APA format exists in a number of resources (Bem, 2004; Calderon & Austin, 2006). Note that outside of parentheses, you use “and” in between authors (or with three-plus authors, “and” just before the last author), but inside of parentheses, you use the ampersand lan66845_02_c02_p023-062.indd 40 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer (&) symbol to connect the authors (or with three-plus authors, just before the last author). Be sure to follow alphabetical order using the first author ’s last name; do not reorder mul – tiple references in parentheses according to year. As you would imagine, there are also detailed rules for the presentations of quotations in text. You would normally use the citation styles here, but in addition, you must report the page number (or the paragraph number). Overall, be sure to follow the instructor ’s preferences for the use of direct quotes in your research paper assignment. If you are going to use direct quotes, be sure to follow the rules of APA format. For instance, if you plan to use a quote of 40 words or more, you must use a block format in your text to set off the block quote. If you omit part of a quote, you must note that by using ellipsis marks (. . .) in the quote; however, be sure that your deletion does not change the meaning of the original idea. If you want to add emphasis to the original quote (for example, by italiciz – ing a word), you must acknowledge that you added the italics to the text (that is, in the original quote what you have italicized was not in italics). If you are quoting from a source with pages, then you must report a page number (e.g., p. 278). If you are reporting from a source without page numbers (for example, a brochure or a website), you must cite the paragraph using the paragraph symbol (e.g., ¶ 7). Preparation Instructions To make your work look its best, I recommend that you follow these instructions when preparing your research paper. Even though you may be delivering your document elec – tronically, you want everything formatted correctly before handing in your Microsoft Word document or PDF. Some of these instructions are for a printed copy, but most apply to either submission mode. • When printing, use bright white paper with crisp black ink. If your ink/toner car – tridge is running low, make sure to use a new one. This work should look its best. • Use the Times New Roman 12 pt font. The only exception would be if you were preparing a figure, and if you are, be sure to follow those APA rules. • Double space everything, including the title page and references. Regarding the preparation of tables, check with your instructor to determine his or her prefer – ence—tables do not have strict line spacing rules in the 6th edition of the Publica- tion Manual (APA, 2010). • Using your word processor (most likely Microsoft Word), set your margins to 1 inch on the left, right, top, and bottom margins. Your page header will go inside the top 1-inch margin. • The sequence of pages in the manuscript follows this order, exactly: title page, Abstract, text (Introduction starting on page 3), References, appendices, foot – notes, tables, figure captions, and figures. Note that you may not have all of these parts in your research paper, but be sure to follow the exact order shown. • Indent your paragraphs 1/2-inch using the tab key, and not hitting the spacebar on your keyboard. • If you are going to use lists, be sure to use them properly. This is called seriation, and there are specific APA rules about how to present lists. There are so many more rules, but those are the main points. For all the details, you’ll want to consult with the Publication Manual (APA, 2010). lan66845_02_c02_p023-062.indd 41 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Common Problems to Avoid You can imagine that with all of these sections, the flow of a research paper might be choppy and the text difficult to read. The skilled writer uses transitions between sec – tions and paragraphs to improve the flow and readability. Here are some suggestions for transitions: • Time links: then, next, after, while, since • Cause–effect links: therefore, consequently, as a result • Addition links: in addition, moreover, furthermore, similarly • Contrast links: but, conversely, nevertheless, however, although, whereas One of the most confusing aspects for a writer who is new to the APA format regards the use of verbs. The verb tense that is used depends upon the section of the paper. When appropriate, use the active voice. Try to increase the frequency of active voice construction: Active: Davis designed the study. Passive: The study was designed by Davis. The passive voice is acceptable when you focus on the outcome of the action, rather than who made the action happen. Try to minimize the use of passive voice: Active: Students administered the survey. Passive: The survey was administered by the students. Use past tense to discuss some – thing that happened at a specific, definite time in the past (e.g., writing about another research – er ’s work or when report – ing your results)—”Landrum (1998) found that 63% of stu – dents reporting average work expected a grade of B or a grade of A.” Use the present perfect tense to discuss a past action that did not occur at a specific, definite time in the past—”Since the completion of the study, we have found further evidence to support our conclusions.” See the following points for more tips about verb tense. Introduction (Literature review) Past tense (“Davis concluded”) Present perfect tense (“Researchers have concluded”) Method Past tense (“Participants completed the task in 5 minutes”) Present perfect tense (“The task was completed by the participants in 5 minutes”) Deciding whether some sections should be written in the active or passive voice can be difficult when using APA format. What are the benefits and drawbacks of each style? iStockphoto/Thinkstock lan66845_02_c02_p023-062.indd 42 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Results Past tense (“Scores declined after the intervention”) Discussion (Discuss outcomes and present conclusions) Present tense (“Participants take the computer task seriously”) The scope of this chapter cannot possibly prepare you for all contingencies with regard to APA format, but the goal is for you to come away with as much practical knowledge as possible. See Table 2.2 for some typical grammatical problems to avoid, with examples of each error (Gottschalk & Hjortshoj, 2004). Table 2.2: Grammatical problems to avoid 20 Common ErrorsExample of the Error No comma after an introductory element. Well it wasn’t really true. Vague pronoun reference. John told his father that his car had been stolen. No comma in compound sentence. I like to eat but I hate to gain weight. Wrong word. His F in math enhanced his alarm about his D in chemistry. Missing comma(s) with a nonrestrictive element. The students who had unsuccessfully concealed their participation in the prank were expelled/compared to/The students, who had unsuccessfully concealed their participation in the prank, were expelled. Wrong or missing verb ending. I use to go often to town. Wrong or missing preposition. Cottonwood Grille is located at Boise. Comma splice. Chloe liked the cat, however, she was allergic to it/compared to/Chloe liked the cat; however, she was allergic to it. Missing or misplaced possessive apostrophe. Student’s backpacks weigh far too much. Unnecessary shift in tense. I was happily watching TV when suddenly my sister attacks me. Unnecessary shift in pronoun. When one is tired, you should sleep. Sentence fragment. He went shopping in the local sports store. An outing he usually enjoyed. /The second part is the fragment./ Wrong tense or verb form. I would not have said that if I thought it would have shocked her. Lack of subject-verb agreement. Having many close friends, especially if you’ve known them for a long time, are a great help in times of trouble. Missing comma in a series. Students eat, sleep and do homework. Lack of agreement between pronoun and antecedent. When someone plagiarizes from material on a website, they are likely to be caught. Unnecessary comma(s) with a restrictive element. The novel, that my teacher assigned, was very boring. Run-on or fused sentence. He loved the seminar he even loved the readings. Dangling or misplaced modifier. After being put to sleep, a small incision is made below the navel. Its/it’s confusion. Its a splendid day for everyone. lan66845_02_c02_p023-062.indd 43 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer In a similar (efficient) vein, here is a very brief reminder about words commonly confused in psychology (Scott, Koch, Scott, & Garrison, 1999)—be sure to get these correct. Spell- checkers and grammar-checkers might not catch these mistakes, so it is up to you to avoid “operator error.” Table 2.3: Commonly confused words advice/advise affect/effect aisle/isle allusion/illusion an/and angel/angle ascent/assent bare/bear brake/break breath/breathe buy/by capital/capitol choose/chose cite/sight/site complement/ compliment conscience/conscious corps/corpse council/counsel dairy/diary desert/dessert device/devise die/dye dominant/dominate elicit/illicit eminent/immanent/ imminent envelop/envelope every day/everyday fair/fare formally/formerly forth/fourth hear/here heard/herd hole/whole human/humane its/it’s know/no later/latter lay/lie lead/led lessen/lesson loose/lose may be/maybe miner-/minor moral/morale of/off passed/past patience/patients peace/piece personal/personnel plain/plane precede/proceed presence/presents principal/principle quiet/quite rain/reign/rein raise/raze reality/realty respectfully/ respectively reverend/reverent A Note About Plagiarism Essentially, plagiarism is when you borrow intellectual property without crediting the original source. There are at least two categories of plagiarism—intentional and unin – tentional (other authors sometimes refer to unintentional plagiarism as sloppy writing; Harris, 2005). Intentional plagiarism means just what it sounds like—cheating on pur – pose. Oftentimes intentional plagiarism occurs due to student procrastination and panic, and the student is under a deadline to complete a writing assignment (Roig, 2008). Some examples of intentional plagiarism are (a) downloading and turning in a paper from the web; (b) including a graph or table from someone else’s work without proper citation; (c) copying phrases, sentences, or paragraphs from others’ work without using proper cita – tion or quotation format; (d) paraphrasing or summarizing others’ work without citation; and (e) turning in your own previously written work when prohibited from doing so by your instructor (Harris, 2005). Roig (2008) referred to this last practice as double-dipping. So to give credit where credit is due, there are three basic strategies: paraphrasing, sum – marizing, and quoting. Paraphrasing is when you take someone’s ideas or words and put them into your own words, and the number of words you use is roughly equivalent to the number of words from the original source. Summarizing is taking someone’s work and putting it into your own words, but the result is shorter than the original (Harris, 2005). A direct quote is just that—using the exact words of the original author. For all three techniques (paraphrasing, summarizing, and quoting), you must cite the work of the originators of the ideas. lan66845_02_c02_p023-062.indd 44 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Unintentional plagiarism can occur through a number of methods. For example, you might not completely understand the rules for citation, you might be careless when you are taking notes, you might be citing opinions from the Internet that are uninformed, or you might be sloppy in fol – lowing APA rules for citation (Harris, 2005). Per – haps you’ve heard the phrase “ignorance of the law is no excuse for breaking the law.” The same principle applies here—just because you may be unaware that you plagiarized does not mean it is OK to do so. The best protection you can have is to know, understand, and apply the rules for proper citation in APA format. Harris (2005) sug – gested this guideline: “If the information came from outside your own head, cite the source” (p. 16). Your instructors and professors can help you make the distinctions between what needs to be cited, and what doesn’t. Be careful with com – mon knowledge as well, because sometimes it is not as common as you think. If the information you want to present as common knowledge truly is common knowledge, then it should not be that difficult to locate a reference citation to support your claim. Although you might not think that plagiarizing is a big deal, it often is. The following listing of reasons why plagiarism is wrong is modified from Appleby (2005, p. 9). • It is considered to be a criminal offense (i.e., the theft of intellectual property) and can result in fines and/or imprisonment. • It is academically dishonest and can lead to serious sanctions from the university. • It undermines the academic integrity and ethical atmosphere of the university. • It violates the mission of higher education to emphasize “a respect for knowledge.” • It involves a passive learning process that obstructs the acquisition and under – standing of meaningful academic material. • It stalls or retards intellectual, moral, and social development. • It is contrary to the concept of critical thinking. • It promotes feelings of lowered self-esteem in those who believe they must prac – tice it to survive academically. • It produces alumni whose inferior knowledge, abilities, and moral standards tarnish the public image of the college and lower the perceived value of a degree in the eyes of those who evaluate current students who are seeking employment or admission into graduate school. • It violates the code of ethics of professional societies that represent psychology. Sometimes it is difficult to differentiate between plagiarism and sloppy citation style, which emphasizes the importance of your instructors’ teaching you about proper citations (in our case, APA style) and how to avoid plagiarism. Let me present you with some of these “sticky situations” and practice a bit with determining whether the writing constitutes The best protection against plagiarism is to know, understand, and apply the proper APA rules for citations. Hemera/Thinkstock lan66845_02_c02_p023-062.indd 45 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer plagiarism or the misuse of sources. The idea for and source of this exercise come from Shadle (2006). Here is the original text from Price (2002), with the proper APA reference: If you were going to use that as a direct quote in your APA-style paper, here is what it would look like (note that the text is indented because the quote is longer than 40 words: But what if a student were to write a paragraph in his or her paper exactly like the one in the box below—would this be plagiarism? For most faculty, the answer would be yes , this is plagiarism. Not only are most of the phrases identical to the original, but there is no attribution to the author—remember, we must give credit where credit is due. As an instructor, if I were to read a paragraph like the one in the box above in a student’s paper, I would have to assume that this idea was the student’s original idea because of the lack of attribution. The previous example is fairly blatant, but what about this one: But plagiarism is not stable. What we think of as plagiarism shifts across historical time periods, across cultures, across workplaces, even across academic disciplines. We need to stop treating plagiarism like a pure moral absolute (“Thou shalt not plagiarize”) and start explaining it in a way that accounts for these shifting features of contexts. Price, M. (2002). Beyond “Gotcha!”: Situating plagiarism in policy and pedagogy. College Composition and Communication, 54, 88–115. Source: Price, M. (2002) But plagiarism is not stable. What we think of as plagiarism shifts across historical time periods, across cultures, across workplaces, even across academic disciplines. We need to stop treating plagiarism like a pure moral absolute (“Thou shalt not plagiarize”) and start explaining it in a way that accounts for these shifting features of contexts” (Price, 2002, p. 90). Source: Price, M. (2002) Plagiarism is very difficult to understand because it is not stable. What we think of as plagiarism shifts across historical time periods, across cultures, across workplaces, even across academic disciplines. We need to stop treating plagiarism like a pure moral absolute and start explaining it in a way that accounts for these shifting features of contexts. Source: Price, M. (2002) According to Price, plagiarism is not stable. What we think of as plagiarism shifts across historical time periods, across cultures, across workplaces, even across academic disciplines. We need to stop treat- ing plagiarism like a pure moral absolute and start explaining it in a way that accounts for these shift – ing features of contexts (“Beyond ‘Gotcha,’” p. 90). Source: Price, M. (2002) lan66845_02_c02_p023-062.indd 46 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer I would consider this example either unintentional plagiarism or just using a sloppy cita – tion method. This example text does give appropriate credit, which is good. However, after the first sentence almost everything else is a direct quote, and thus should be pre – sented as a direct quote. This example is using Modern Language Association (MLA) cita – tion style, not APA. Make sure you follow the style that your instructor wants, not a style you may have previously learned in another class or institution. Sometimes students become frustrated with the necessity for citation, and when they look at their completed APA-style manuscripts, they see citations all over the place. They won – der where the creativity is in all of this if their research paper is about everyone else’s ideas. The creativity in psychology is in the combination of ideas—that is, how you put the ideas together. When you see trends in the literature or you identify common threads across different areas of social science, that’s creative. The creativity comes in the com – binations of new ideas, or the development of a new method to test a hypothesis, or an innovative approach to understanding an age-old problem. Psychologists are creative, but we also value the intellectual property of others, which is why we are so careful to avoid plagiarism. Roig (2008) articulated the importance of ethical writing this way: “clear and effective writing is critical to academic success, and it is one of the most valued skills in the modern workplace. However, whether it is being used for academic or professional purposes, writing must not only be mechanically sound, clear, and persuasive, it must also be accurate and, above all, honest” (p. 33). There are strategies that you should follow that will help you avoid a charge of plagia – rism. The strategies in the list below come from Harris (2005), and they are excellent sug – gestions for protecting yourself from plagiarism. • Protect your data and your computer passwords to protect against theft. • Do not lend, give, or upload any paper—even if a student just wants to “see” what an APA-formatted paper looks like. • Report any theft immediately, including the proper authorities, and in the case of your academic work, your instructors. • Save and print all drafts and notes—having these items will help support the originality of your written work. • Photocopy or print all of your sources—and do not cite something that you have not actually read yourself. • Be proactive in seeking out the advice of your instructor and/or teaching assis – tants. If someone has been reviewing your work all semester, it will be easier for you to make the case that your work is actually your work. Plagiarism is a form of cheating with serious consequences. Many agree with Harris (2005) when he states “the goal of education is not to get through, but to get better” (p. 15). If you intend on cheating your way through college, why bother? Would you want to have life-saving surgery performed by a surgeon who cheated his or her way through medical school? Would you want to consult a lawyer who cheated through law school, or a thera – pist who cheated through graduate school? Plagiarism and cheating have the potential to be harmful to others, but most of all, you hurt you. lan66845_02_c02_p023-062.indd 47 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Project Checklist: Quick Reminders About APA Style and Format This quick summary will help you remember the key elements of preparing papers in accordance with APA format. As always, heed your instructor’s departures from these rules as needed. Title Page• Page header inside 1-inch margin, number every page. • Running head inside top margin on every page; actual running head in CAPS. Limited to 50 char – acters. Page 1 header format is different from page header format on subsequent pages. • Title, name, and affiliation block are horizontally and vertically centered. • Everything double-spaced. • Title is no longer than 12 words. • Author underneath title, author affiliation underneath author. • In the end, your paper should look like the example paper starting on p. 41 of the Publication Manual (APA, 2010), except you will probably not have author notes. Abstract • Starts on its own page (page 2), Abstract centered on line 1. • Everything double-spaced; Abstract paragraph not indented. • No longer than 120 words. Body of Manuscript • Starts on its own page (page 3). • Repeat the exact title from page 1 at the top of page 3. • Everything double-spaced. References • Starts on its own new page after the end of the Discussion section text. • Everything double-spaced. • Every reference cited in the manuscript should be in the reference section. Every reference in the reference section should be in the manuscript. All name spellings should match. • References are prepared following capitalization guidelines, italicizing rules, indentation, etc. • References are important. They show off your academic achievement and your grasp of the liter – ature. They provide guidance to those who want to read what you have read. This is your chance to show your scholarly abilities. General Comments • No extra spacing between paragraphs—regular double-spacing throughout paper. • One-inch margins on all four sides of the paper. • No right justification (ragged right margins). • References cited correctly in manuscript. Mostly paraphrase. • When quoting, use proper format; avoid plagiarism. • No contractions, no abbreviations (unless APA approved). • Write in complete sentences. Avoid awkward constructions. Avoid being too colloquial (too infor – mal). Your paper is not a conversation between student and instructor. • Hand in paper as PDF, Microsoft Word file, or however you are instructed to turn in your work. If printing a paper copy, print using crisp black ink on bright white paper. No printer problems (e.g., faded print, smudged ink). Staple once in upper left corner. Use the Times New Roman font with no changes in font or font size; use 12 point font throughout. • Ask if you have questions. lan66845_02_c02_p023-062.indd 48 4/20/12 2:44 PM CHAPTER 2 Section 2.1 Writing Scientifically: A Brief Primer Classic Studies in Psychology: John B. Watson and Little Albert One of the themes of this book is that psychologists strive to be good storytellers. That is, a good story has compelling characters, some sort of conflict, escalation of and a resolution to that conflict, and per – haps even a moral to the story. The story of Little Albert is a compelling story, and one that has been retold over and over in introductory psychology textbooks. It’s a fascinating story; let’s examine why. In 1920 John B. Watson (often credited as the founder of behaviorism) and Rosalie Rayner (a gradu – ate student working with Watson at the time) published a detailed account of their work with a young child, Albert B. (interestingly, Watson and Rayner never call Albert by Little Albert in the 1920 article; this must be an affectation added later by someone else). When the research began, Albert was approximately 9 months old. The goal of Watson and Rayner (1920) was to determine if they could instill in Albert a conditioned emotional reaction; that is, could they make Albert learn to have an emotional response to a stimulus that was previously neutral? In other words, this was a test to see if Albert could become classically conditioned to have a fear response. If you read the original 1920 article and compare it to today’s stan – dard of methodology used by researchers studying classical condition – ing and learning, the procedures that Watson and Rayner used are muddled at best. First, they showed Albert neutral stimuli that they hoped would become conditioned stimuli: a white rat, a rabbit, a dog, a monkey, masks with and without hair, cotton wool, etc. Albert’s mother, hospital attendants, and the researchers were on hand and reported that mere exposure to these items did not cause Albert to exhibit fear or rage. Next, a 4-foot steel bar 3/4 of an inch in diameter was struck with a hammer to make a loud sound right behind Albert. This sudden and unexpected noise was followed by Albert crying. This is the unconditioned stimulus (loud noise) being paired with the unconditioned response (crying). The question for Watson and Rayner was this: Can pairing an originally neutral object (the CS—a rat, a rabbit, etc.) with the loud sound (the UCS) eventually lead to the CS’s eliciting the CR (a fear response)? Quite honestly, the answer is muddled. So many different variations were tested that it is difficult to determine if a conditioned emotional response was learned by Albert. Multiple pairings of the CS and UCS were presented, but sometimes Albert played with blocks or sucked his thumb, and those behav – iors may have played a role in the patterns that he developed. Watson and Rayner were interested in whether this type of learning would last over time, and acknowledged that the outcomes of the study might be detrimental to Albert. Yet, Albert left the hospital where he was being tested before the series of research studies were complete. I have tried to tell this story as accurately as I could, referring back to the original 1920 Watson and Rayner publication. This research serves as a famous story in psychology, and it may provide some insight as to appropriate and inappropriate use of methodology and how those who cannot protect themselves (in this case, a child) must be protected by the ethical standards of psychologists. How – ever, the results that Watson and Rayner (1920) concluded have not stood the test of time, although the “Little Albert” story has been told and retold erroneously through the years (Harris, 1979). In fact, some of those accused by Harris of getting the facts wrong about Watson and Rayner have accused Harris (1979) of getting his facts wrong (Seligman, 1980). “Little Albert” makes for a great story (Paul & Blumenthal, 1989), but as you read about past research as you write your own literature review, if at all possible, consult with the original source material, because even your textbooks can get a story wrong, and then the story can take on its own apocryphal life (Harris, 1979). It’s OK to tell a good story, but it’s even more important to get the facts straight. Even so, Watson’s life makes for a good story, from founding behaviorism in psychology to becoming a major player in the world of advertis – ing. There are also some interesting soap-opera-like twists and spins to his life as well. If Corbis/AP Images (continued) lan66845_02_c02_p023-062.indd 49 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns 2.2 Ethical Concerns I n psychology, “we” are the subject of study. When I say we, I mean human beings (for the most part). Sometimes animals are the subject of study, but then again, technically we’re animals too. When human beings provide the data for our studies, this compli – cates how scientific research is conducted and often makes for an ethically complex situ – ation. There are fundamental principles that we follow regarding ethics and our code of conduct in psychology. The Ethics of Research with Humans and Animals Psychologists have been concerned about ethics for some time, although it took some time for these interests to become formalized. The typical “start date” for psychology as a discipline is 1879 in Germany, with the American Psychological Association being founded in 1892. In 1938, APA formed the Committee on Scientific and Professional Ethics (before that time, ethics complaints were handled by existing APA standards) (Hanson, Guenther, Kerkhoff, & Liss, 2000). With the growth in psychology during and following World War II, in 1947 the “Code Governing the Professional Practice of Psychology” was adopted. Interest in the ethical code and the importance of its use continued to grow over time. Part of the motivation for the current protections of human subjects comes from the abuses of German scientists during the Nazi era, but abuses are not limited to that particular era. American scientists conducted the Tuskegee syphilis study in which they “recruited poor black southern men with syphilis for a longitudinal study of the course of the disease” (Singer & Levine, 2003, p. 149). Inexplicably, even after the discovery of peni – cillin, these men were not informed about this new and effective treatment. The Tuskegee syphilis study lasted for 40 years, and African-American men with syphilis (“bad blood”) you are interested in learning more about Watson’s life and contributions, I highly recommend The Mechanical Man by Buckley (1989). For an update into the pursuit of the actual identity of Little Albert and what happened to him after laboratory experiences, see the interesting investigative story by Beck, Levinson, and Irons (2009)—although you should know that the story does not have a happy ending, nor do some think this is the entire story, and that parts of the mystery are yet to be solved. Critical Thinking Questions: 1. Why do you think the story of Little Albert remains to be a popular and engaging story to this day? What are the characteristics of good stories that make them so memorable? To what extent can you utilize these storytelling strategies and tactics for your applied project? 2. With the research conducted here, there is no mention of an Ethics Board or IRB (more about these topics soon). What type of ethical concerns might a person have about this type of research? Who should have the responsibility for conducting this type of research? If this research were deemed too dangerous to conduct, who is to make that decision, and how might it be enforced? 3. Sometimes academics snipe at one another in the literature, such as alluded to in the Little Albert research and subsequent publications. What do you think about that? Why might psychologists be so passionate about this topic that they would criticize each others’ work in print? Is this an appropriate strategy? How might differences in drawing conclusions be resolved when research- ers disagree? Classic Studies in Psychology: John B. Watson and Little Albert (continued) lan66845_02_c02_p023-062.indd 50 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns went untreated for syphilis until death, when their bodies were autopsied. The behavior of the scientists here was reprehensible. A poorly designed and implemented study need – lessly perpetuated human suffering for decades. Other research, more of a psychological nature (and usually involving deception), accelerated society’s interest in the protection of human subjects, particularly the Milgram obedience to authority studies of the 1960s and 1970s. See the case study at the end of this section for more on the Milgram study. The first ethics code for the American Psychological Asso – ciation was adopted in 1952 (Hanson et al., 2000), and the current revision was codified in 2002 (you can view the entire ethics code at www.apa.org/ ethics ). Between the biomedi – cal abuses of the Nazis and the Tuskegee syphilis study, and psychology pushing the boundaries via Milgram’s study of obedience to author – ity, more protections for human subjects were called for. The National Research Act of 1974 was enacted to establish human research protections and ensure the rights of participants in both biomedical and behavioral research (Singer & Levine, 2003). This led colleges and universities to develop local Institutional Review Boards (IRBs) to aid in the protection of human subjects. The National Research Act of 1974 also created a board to study these issues, and that board published its report (called the Belmont Report) in 1979. Various rulings from federal agencies occurred over the years, and even – tually all of the rules and previous laws were brought together into the Code of Federal Regulations for the Protection of Human Subjects—known as 45 CFR 46 (Singer & Levine, 2003). These federal regulations address the three main ethical principles found in the Belmont Report: beneficence, respect for persons, and justice. Beneficence is the idea that the potential harm that research participants may experience must be balanced by the potential benefits of the research. Respect for persons led to the requirement of informed consent; that is, human participants deserve to know the risks involved in research and what their protections are. Justice is the idea that the burden of research does not fall exclusively on any one group or class of individuals in society (Singer & Levine, 2003). Thus, the guidelines established at the federal level are enforced at the local level through the Institutional Review Board . The benefits of participation as well as the associated risks should be equally distributed across participants. Although humans are the primary species of interest for most psychologists, studying animal behavior can be a very effective means for gaining insight into human behav – ior. Additionally, with proper controls, experiments can be conducted on animals that cannot be ethically conducted on humans. Some people find this practice objectionable and believe that animals should not be used in experiments of any kind, even if no pain The ethics violations of the Tuskegee syphilis studies were so reprehensible that they helped spawn a higher ethical code. In 1997, President Clinton even issued a formal apology to survivors. Associated Press lan66845_02_c02_p023-062.indd 51 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns is involved (which is often the case in animal studies). How – ever, most of the pharmaceutical breakthroughs that we appreci – ate today involved earlier clini – cal trials with animals prior to human clinical trials. From a more psychological perspective, many of the principles that we have come to understand about human behavior were first understood via animal behavior. Also, psychologists are bound by ethical codes to minimize pain and provide comfort and shelter to animals being stud – ied. At colleges and universities, research with animals is con – ducted within the regulations of the Institutional Animal Care and Use Committee, and many of these guidelines have been codified into law in the Animal Welfare Act of 2002 (Ator, 2005). Animals, in many respects, provide us with good models of human behavior; these practices are not supported by all, however. The IRB and the Role of Informed Consent Prior to any research being conducted, approval is required from the appropriate body at your institution. At many universities, this responsibility for monitoring and approving research with human subjects is the IRB (at smaller schools without an IRB, these reviews are typically done by a faculty member or a departmental committee). For work with ani – mals, typically research protocols would be reviewed by an Institutional Animal Care and Use Committee (IACUC). But what is the overall purpose of the IRB process? Ultimately, it is the protection of human participants who participate in research. “Informed consent is designed to protect subjects and ensure their autonomy” (Agre & Rapkin, 2003, p. 1). To be protected, human participants in research need to understand the basic elements of what is going to happen during the research process. At a minimum, participants need to be told about the researchers and the nature of the research, the risks and benefits of participation, who will be able to access the information participants’ provide, the right to withdraw, any costs or compensation they will receive, and the responsible party other than the researchers (typically, this would be the IRB of the college, university, or agency) (Binik, Mah, & Kiesler, 1999). Informed consent is an essential component of the research process because it helps to protect the participant’s rights. Anonymity, Confidentiality, and Debriefing Anonymity and confidentiality are two additional concepts that researchers need to be sensitive to when conducting research. Anonymity refers to the absence of a connection Animals are used in testing because some experiments can be ethically performed on animals, but not on humans. Do you think that animal testing is ethical? Why or why not? Photodisc/Thinkstock lan66845_02_c02_p023-062.indd 52 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns between a specific participant and the data that he or she provides. Data collection records are anonymous when there can be no link between a specific individual and the data he or she provided. This is usually achieved by telling participants not to put their name or any identifying number (for example, student ID number or social security number) anywhere when responding in the study. However, sometimes anonymity is not possible. For example, participants in a study may agree to be videotaped. Given the nature of the recording, it is virtually impossible to guarantee anonymity because their identity is inher – ently linked to the data; their visual images compose the data. By providing the protec – tion of anonymity to participants, we are protecting their privacy. The privacy protections provided to participants are directly related to the underlying expectations of beneficence and trust (Folkman, 2000). However, protecting privacy and providing anonymity can be challenging, especially with the emergence of the Internet as a means of data collection (Nosek, Banaji, & Greenwald, 2002). Confidentiality differs from anonymity in that confidentiality refers to the experiment – er ’s promise not to reveal the results from a particular individual unless that individual explicitly allows the experimenter to do so. In other words, the results of any one par – ticipant are held in confidence with the researchers, and they promise not to reveal spe – cific information. If for some reason the researchers desire to identify a particular person with his or her data, they must acquire written consent from the participant. Most of the time confidentiality is not an issue because researchers are not interested in particular individuals but rather the performance of a group of individuals. If individual data are important, researchers can use codes to protect the identity of the participant while still communicating the data of an individual. Debriefing is a process that occurs at the conclusion of a study. The history of debriefing has its roots in military campaigns (Lederman, 1992), where individuals who were not present at an event informed others as to what happened. In a psychological context, debriefing involves informing participants of the actual events that have just occurred, especially if deception was involved. Debriefing provides the opportunity to inform, educate, check on methods used, and undo negative consequences if necessary. When deception is used in psychology, the debriefing also serves as a dehoaxing—that is, let – ting the participants know fully about the deception that was used during the study (Lederman, 1992). Note that the use of deception is not inherently evil, but deception must be used judiciously and benefits must greatly outweigh the risks. If you want to study topics like altruism, obedience to authority, or motivation, then the use of decep – tion (with IRB approval) may be the route to go to understand the concept of interest. For example, if you are interested in humility, rather than ask the survey question “Are you humble?” you could design a situation where humble behavior could be observed (rather than rely on self-report). If you were to conduct a study using deception, the debriefing would typically consist of three elements: you would tell the participant about the nature of the deception, the true purpose of the experiment, and the reasons why the deception was necessary (Lederman, 1992). In a study about the effectiveness of debriefing, Brody, Gluck, and Aragon (2000) found that the most common problems with debriefings are that they are unclear or that more information was desired by participants. The next most frequently reported nega – tive outcome of debriefings is that they were short. What does this mean to you as the researcher? The debriefing portion of research is important to ensure that participants lan66845_02_c02_p023-062.indd 53 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns can gain as much as possible from your study. When designing the study, make sure that you include details about your hypothesis and what you hope to find, as well as leave enough time in the experimental session to answer participants’ questions. The debrief – ing, in some ways, may be the most valuable part of the research experience for those students completing research toward course credit. Conducting research is a complicated enterprise, not only from a research methods point of view but also from an ethical perspective. As psychologists we have an utmost respon – sibility to protect the health and welfare of our participants, and at the same time pursue worthy research projects that enable us to test our hypotheses (both scientific validity and scientific value). A psychologist must never take lightly the consideration of using humans or animals for research purposes, and the potential benefits from such research enter – prises must always outweigh any potential costs or harms to the participant. These beliefs are reiterated in the following subsections on the rights and responsibilities of research participants (Korn, 1988). Finally, ethical decisions are not made in a vacuum, but can be politically charged at times (Baarts, 2009). Human behavior is complex, and humans behaving ethically within context can mean that scientists must discuss and sometimes debate the proper actions and procedures of science. These discussions and debates are healthy and should continue to occur publicly so that our collective wisdom about ethical research and ethical behavior continues to grow. Rights of Research Participants Respecting the rights of research participants is a vital part of any study. Participants should expect the following: • Participants should know the general purpose of the study and what they will be expected to do. Beyond this, they should be told everything a reasonable person would want to know in order to decide whether to participate. • Participants have the right to withdraw from a study at any time after beginning participation in the research. A participant who chooses to withdraw has the right to receive whatever benefits were promised. • Participants should expect to receive benefits that outweigh the costs or risks involved. To achieve the educational benefit, participants have the right to ask questions and receive clear, honest answers. When participants do not receive what was promised, they have the right to remove their data from the study. • Participants have the right to expect that anything done or said during their participation in a study will remain anonymous and confidential, unless they specifically agree to give up this right. • Participants have the right to decline to participate in any study and may not be coerced into research. When learning about research is a course requirement, an equivalent alternative to participation should be available. • Participants have a right to know when they have been deceived in a study and why the deception was used. If the deception seems unreasonable, participants have the right to withhold their data. • When any of these rights is violated or participants object to anything about a study, they have the right and the responsibility to inform appropriate university officials, including the chairperson of psychology department and the Institu – tional Review Board. lan66845_02_c02_p023-062.indd 54 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns Responsibilities of Research Participants In addition to rights, research participants also have responsibilities of their own, includ – ing the following: • Participants have the responsibility to listen carefully to the experimenter and ask questions in order to understand the research. • Participants should be on time for the research appointment. • Participants should take the research seriously and cooperate with the experimenter. • When the study has been completed, participants share the responsibility for understanding what happened. • Participants have the responsibility for honoring the researcher ’s request that they not discuss the study with anyone else who might be a participant. You can view the Ethical Standards and Code of Conduct at www.apa.org/ethics . You might be surprised to find that the Ethics Code also addresses teacher and student issues as well. Case Study: Stanley Milgram and Obedience to Authority When you get the chance, read the original Milgram (1963) “Behavioral Study of Obedience.” Not only is it fascinating for the work it describes, but it is also well written and truly a classic on many levels. Starting in the late 1950s in Norway and France, and continuing at Princeton, Yale, and eventually Har – vard, Stanley Milgram completed a series of systematic studies that examined obedience to authority. Although obedience can be productive and is necessary for a civilized society, obedience to authority can and has been abused. Milgram (1963) said it more eloquently: “obedience may be ennobling and educative and refer to acts of charity and kindness, as well as to destruction” (p. 371). Part of Mil – gram’s interest in obedience to authority was to understand the behavior of soldiers leading up to and during World War II who carried out atrocities against millions of innocent people. Even if one or two people were the masterminds of such evil acts, those acts could not be accomplished without the help of many who were obedient to authority. To better understand the conditions by which obedience occurs, Milgram used a laboratory on the grounds of Yale University to conduct a study of “learning and memory.” He advertised in the local New Haven (Connecticut) newspaper for participants—they were paid $4.50 and were skilled and unskilled workers, salesmen and businessmen, and professionals (in this particular study they were all men). On the day of the experiment, two participants showed up to the laboratory, although one of these “participants” was part of the study—this person is called a confederate. Each of the partici- pants drew a slip of paper to determine who would be the teacher and who would be the learner in the learning and memory study, but this was rigged as well; the actual participant was the teacher, and the confederate was always the learner. The supposed purpose of the experiment was to determine how effective the delivery of punishment would be in helping someone learn word pairs. As it would turn out, the learners in Milgram’s studies weren’t very good learners at all—of course, the point of the study was to determine how much punishment the teacher would deliver when told by an author – ity figure to do so. Before learning word pairs, the learner was strapped into a chair in an adjacent room (many variables were varied across multiple studies—see Milgram (1965) for more variations). To test to see if the equipment was working (and to convince the teacher about the delivery of shock), both the teacher and the learner received a 45-volt shock when the 45-volt shock lever was switched (note that this was the only time in the experiment when actual shocks were delivered). When data (continued) lan66845_02_c02_p023-062.indd 55 4/20/12 2:44 PM CHAPTER 2 Section 2.2 Ethical Concerns collection began, no actual shocks were delivered. The teacher taught word pairs to the learner, and then the teacher stated one of the words in the pair and the learner was to respond with the other word. On purpose, the learner was bad at this task, and the teacher was instructed to flip a switch on a shock generator each time the learner made a mistake (although, remember, no actual shocks were delivered). There were 30 shock switches on the apparatus, labeled from 15 volts to 450 volts in 15 volt incre- ments. Thus, the dependent variable for each teacher was the number of shocks that they were willing to deliver over the course of the “learning and memory” study. So for each mistake, a shock was deliv – ered, and on the next mistake the teacher would deliver the next (higher) voltage level. The question was how high a voltage would a teacher deliver in this situation? That is, how obedient to authority would the participant be in delivering electric shocks? Milgram (1963) was methodologically very rigorous in his approach. During the study, he was prepared for teachers to ask to stop the study or to stop delivering shocks. The experimenter in the room, often dressed in a white lab coat, said the following “prods” each time the teacher expressed a willingness to stop: Prod 1: Please continue or please go on. Prod 2: The experiment requires that you continue. Prod 3: It is absolutely essential that you continue. Prod 4: You have no other choice, you must go on. Remember that although the learner was strapped in, the teacher was not. The teacher could stop at any time. If the teacher asked about permanent injuries to the learner, the experimenter replied that “although shocks may be painful, there is no permanent tissue damage, so please go on.” So to what degree would the teacher shock the learner, who was not very good at learning word pairs? First, you need to realize how realistic and dramatic this experiment was. Below is a direct quote that describes it in some detail (Milgram, 1965). Also note that in this quote, the victim is the learner—the confederate who is receiving the “electric shocks.” The victim indicates no discomfort until the 75-volt shock is administered, at which time there is a light grunt in response to the punishment. Similar reactions follow the 90- and 105-volt shocks, and at 120 volts the victim shouts to the experimenter that the shocks are becoming painful. Painful groans are heard on administration of the 135-volt shock, and at 150 volts the victim cries out, “Experimenter, get me out of here! I won’t be in the experiment anymore! I refuse to go on!” Cries of this type continue with generally rising intensity, so that at 180 volts the victim cries out, “I can’t stand the pain,” and by 270 volts his response to the shock is definitely an agonized scream. Throughout, he insists that he be let out of the experiment. At 300 volts the victim shouts in desperation that he will no longer provide answers to the memory test, and at 315 volts, after a violent scream, he reaffirms with vehemence that he is no longer a participant. From this point on, he pro – vides no answers, but shrieks in agony whenever a shock is administered; this continues through 450 volts. (p. 62). Of the 40 men in the Milgram (1963) study, how many went on to deliver the 30th shock at 450 volts? In this study, 26 out of 40, or 65%, delivered the full number of shocks to the learner. After the study was complete, the learner came out to meet the teacher, and the teacher was assured that no actual shocks were delivered. The teacher was debriefed and dehoaxed about the deception Hulton Archive/Getty Images Case Study: Stanley Milgram and Obedience to Authority (continued) (continued) lan66845_02_c02_p023-062.indd 56 4/20/12 2:44 PM CHAPTER 2 Chapter Summary Chapter Summary I n this chapter two very practical matters were presented that affect a variety of types of research projects within psychology—scientific writing and the protection of human participants and animal subjects in research studies. These practical matters are pre – sented early in this book because they apply to all research designs. If you have completed the most inspiring and valuable research ever conducted, but you fail to communicate it adequately following the standard reporting mechanisms of science, it will be difficult for your research to be taken seriously by other scientists. Developing the skills of good sci – entific writing—precision, accuracy, the ability to identify an underlying thread or theme, analysis, synthesis—will serve you well in the future. Similarly, the advances we make in understanding our world from a psychological perspective typically come via the efforts of the participation of others. Protections of others must be ensured so that no harm (or minimal harm) comes to those that help us advance the science of human behavior. There are examples scattered throughout the history of the sciences where abuses of individuals and groups have occurred, and we must guard against those potential abuses vigilantly; advancing our understanding of human behavior must not occur at the expense of the torture or humiliation of those who we seek to understand. used in the experiment. Milgram followed up with a sample of participants to make sure they were OK one year later, and they were. Milgram conducted many variations of this study, such as the loca- tion where the study was conducted, how much access the teacher had to the learner, and testing with individuals versus groups (Milgram, 1965). Although the percentages varied (not always 65% who shock to 450 volts), the percentages were higher than typically expected, especially when experts were consulted (Milgram, 1965). These studies provided important insights into obedience to authority, such as the events of World War II. In fact, in a review of Milgram’s work, Packer (2008) determined that these findings were relevant to the treatment of prisoners, helping to understand situations from the atrocities of the Holocaust to torturing prisoners at Abu Ghraib. Milgram’s legacy and influence in social psychology continues to be strong to this day (Benjamin & Simpson, 2009). When thinking about these results, we might like to assure ourselves that we would not act as those participants did in the 1960s; however, Burger (2009) recently completed a partial replication of the Milgram obedience to authority study and found comparable percentages of individuals willing to administer shocks. Reflection Questions: 1. Certain studies in psychology emerge as key, central works in the field—these are called “seminal” works. The Milgram studies are certainly seminal works, but why? Other researchers must have certainly studied topics such as obedience to authority prior to Milgram, so what makes his work so vital to the field? 2. Topics like obedience to authority in social psychology can sometimes highlight the best and worst in all of us. What do you think the internal influences were that governed a small minority of par – ticipants to discontinue “shocks” early on? What would you do? Perhaps more importantly, why are we (in general) so poor at predicting our own behavior as well as the behavior of others? 3. What do you think about the use of a confederate in the study? What steps would the research – ers need to follow in accordance with IRB and ethics guidelines to utilize a confederate, and what would need to occur at the conclusion of the study regarding deception? Case Study: Stanley Milgram and Obedience to Authority (continued) lan66845_02_c02_p023-062.indd 57 4/20/12 2:44 PM CHAPTER 2 Concept Check Concept Check 1. According to the APA style, the method section should appear immediately after which section? A. Results B. Abstract C. Introduction D. Discussion 2. Howard described the content of the surveys used to collect data from his partici – pants. Where should Howard place this description in the method section of his research paper? A. Procedure B. Materials C. Results D. Participants 3. According to Plonsky (2006), the results section should A. organize findings by variable or hypothesis. B. discuss which aspects of the research were proven. C. state the alpha level and null hypotheses. D. explain the implications of the outcomes. 4. In APA style, the physical format includes A. Courier New font. B. 1-inch margins all around. C. single spacing. D. page numbers on the bottom of each page. 5. Anonymity refers to A. informing deceived participants about the actual purpose or events of the study. B. failure to reveal individual data or results. C. the research protocol reviewed by an Institutional Review Board. D. the inability to connect participants to their individual data. Answers 1. C. Introduction. The answer can be found in Section 2.1. 2. B. Materials. The answer can be found in Section 2.1. 3. A. Organize findings by variable or hypothesis. The answer can be found in Section 2.1. 4. B. 1-inch margins all around. The answer can be found in Section 2.1. 5. D. The inability to connect participants to their individual data. The answer can be found in Section 2.2. lan66845_02_c02_p023-062.indd 58 4/20/12 2:44 PM CHAPTER 2 Key Terms to Remember Questions for Critical Thinking 1. Sometimes students who are participating in a research study might think of themselves as “guinea pigs”—and sometimes even researchers in training (such as undergraduate psychology majors) might cast the same aspersions. If you were to overhear this type of conversation in a research context, what would you say? 2. If the ability to write in APA style is so important, then why are so many other style guides available (e.g., MLA, Turabian, Chicago Manual of Style)? Why shouldn’t all undergraduates just learn to write in APA style? 3. In considering past research, it is sometimes easier to see when physical harms are occurring as opposed to psychological harms. If you were conducting research on a psychologically sensitive topic, what steps might you need to fol – low to ensure that no psychological harm was occurring? In other words, how do you know if psychological harm is occurring during a research study? If you were to discover the occurrence of psychological harm, what would be your next steps as a researcher? Key Terms to Remember abstract A quick synopsis of the main points of a research paper. APA style limits the length of an abstract to 120 words. anonymity The absence of a connection between a specific participant and the data that he or she provides. APA style The writing style utilized for social science research results as outlined by the American Psychological Asso – ciation. This writing attempts to com – municate objectivity, credibility, and an evidence-based approach. author notes A portion of the research paper located on the first page outlining specific notes or affiliations that an author wishes to reveal. beneficence An ethical principle found in the Belmont Report that states the poten – tial harm that research participants may experience must be balanced by the poten – tial benefits of the research. communicability Communication of scientific, psychological knowledge by the reporting of results in a consistent and predictable format. confederate An individual who is part of a research study who acts as a participant during the research project. confidentiality The experimenter ’s prom – ise not to reveal the results from a par – ticular individual unless that individual explicitly allows the experimenter to do so. debriefing A process that occurs at the conclusion of a study that informs par – ticipants of the actual events that have occurred during the study, especially if deception was involved. doi A digital object identifier code that is now used on some resources being pub – lished into the literature. The doi code pro – vides a unique numerical identifier of the permanent location of the electronic file on the Internet, primarily journal articles. lan66845_02_c02_p023-062.indd 59 4/20/12 2:44 PM CHAPTER 2 Key Terms to Remember ethics The outlined principles that are fol – lowed by a given group or organization to uphold a moral code of conduct. Institutional Review Board (IRB) Institu- tions that approve and investigate research studies and protocols to ensure ethical consideration is given to the protection of human subjects. intentional plagiarism Purposeful and intentional cheating often due to procrasti- nation and panic about assignment dead – lines. Not giving credit for ideas and work that has been previously produced from other sources. introduction The portion of a research paper that provides the reader with a context for everything that is going to be investigated. This includes introducing the research problem, developing the background, and stating the purpose and rationale for the research paper, including hypotheses. justice An ethical principle found in the Belmont Report that the burden of research does not fall exclusively on any one group or class of individuals in society. materials A portion of a research paper included in the method section that pro – vides the details of the actual items or objects that were used to carry out the study. method The section of a research paper that outlines how the study was conducted so that other researchers can replicate your study. The subsections include partici – pants, materials, and procedure. participants The portion of a research paper included in the method section that describes the characteristics of the indi- viduals who completed your study. plagiarism When you borrow intellectual property without crediting the original source. See intentional plagiarism and unintentional plagiarism. procedure The portion of the research paper included in the method section that guides the reader through the process you used to conduct your research, step by step in chronological order. references The section of the research paper that contains a listing of every cita – tion that you used in the paper. replication The ability to repeat a study. respect for persons An ethical principle found in the Belmont Report that led to the requirement of informed consent; that is, human participants deserve to know the risks involved in research and what their protections are. results The section of the research paper that tells the reader the outcomes of the study, typically from a quantitative or qualitative viewpoint. This section pres – ents the data; it does not interpret them. running head A heading in the research paper that appears at the top right-hand corner with the first five characters of the paper title in upper case letters. The words “Running head” only appear on the first page of the paper. unintentional plagiarism When citation guidelines are not strictly followed to give credit where credit is due. Typically occurs as a result of careless note-taking practices, misunderstanding of citation rules, citing uninformed opinions, or following APA rules for citation in a sloppy manner. lan66845_02_c02_p023-062.indd 60 4/20/12 2:44 PM CHAPTER 2 Web Resources Web Resources American Psychological Associations ethics page. This page discusses ethics codes and promotes ethical responsibility of psychological researchers. http://www.apa.org/ethics/ Official Website of the Collaborative Institutional Training Initiative. This is a resource researchers can use to learn about the ethical responsibility they have in conducting research and receive certification. htt ps://www.citiprogram.org/Default.asp How to avoid plagiarism and how to determine when and where to give credit to a pre – vious researcher and their work. This website also assists researchers in differentiating common knowledge versus something that needs to be cited. http://owl.english.purdue.edu/owl/resource/589/02/ American Psychological Association’s formatting rules. It guides researchers to proper resources concerning APA style and presents an array of products a researcher can use to assist with writing academic research. http://www.apastyle.org/ lan66845_02_c02_p023-062.indd 61 4/20/12 2:44 PM lan66845_02_c02_p023-062.indd 62 4/20/12 2:44 PM 3 Between and Within Groups Research Designs Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • understand and articulate the four basic building blocks of research designs (pretest-posttest or posttest only; one independent variable or more; between, within, or mixed design; and randomization method). • describe the advantages and disadvantages of a between groups design, recognizing the importance of main effects and interactions. • appreciate the special challenges and approaches used in between groups designs, such as demand characteristics and single-blind and double-blind experiments. • recognize the characteristics of within groups research designs (split plot, repeated measures) and understand the principles applied in using within groups designs. • comprehend the special considerations of within group design usage (ceiling and floor effects, carryover and order effects) and appreciate the limitations on any experimental design. Digital Vision/Thinkstock lan66845_03_c03_p063-100.indd 63 4/20/12 2:47 PM CHAPTER 3 Introduction Introduction I n research, the type of design that you choose influences the type of conclusion that can be drawn from the research. It is important to evaluate the needs and circum- stances of your research project before selecting a design. We’ll begin this chapter with a basic overview of the major types of research designs, and then we’ll focus the rest of this chapter on between and within groups designs. Voices from the Workplace Your name: Shelley D. Your age: 46 Your gender: Female Your primary job title: Residential Care Facility Administrator Your current employer: Hope Haven Area Development Center Corporation How long have you been employed in your present position? 3 months What year did you graduate with your bachelor’s degree in psychology? 1983 Describe your major job duties and responsibilities. I oversee a 15 bed residential care facility. This includes supervising a staff of 14, being responsible for the budget of the facility, making sure all policies and procedures are followed according to local, state, and national guidelines. What elements of your undergraduate training in psychology do you use in your work? Frequently look at diagnosis and medications. Have to be aware of what medications are used for what mental illnesses, what symptoms of various mental illnesses are and how to interpret IQ testing. What do you like most about your job? I enjoy working with the residents and assisting them in obtaining various skills in order to reach their potential. What do you like least about your job? The constantly changing regulations. Beyond your bachelor’s degree, what additional education and/or specialized training have you received? I have ongoing training through our organization. This includes CPR and first aid, Adult Abuse report – ing, various leadership and supervisory training, Employment Specialist training. What is the compensation package for an entry-level position in your occupation? We offer health and dental insurance as well as paid life insurance. We have a retirement program in which Hope Haven will provide matching funds up to 3% of your annual salary. We also offer paid sick, vacation and casual days. What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? These are generally the common and standard benefits available in this area to entry level positions in this field. (continued) lan66845_03_c03_p063-100.indd 64 4/20/12 2:47 PM CHAPTER 3 Section 3.1 The Basic Components of Research Designs What are the key skills necessary for you to succeed in your career? Being good with people, good communication skills both verbal and written, ability to use good judg- ment, be a self starter, good organization skills, good typing skills and flexibility. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? Abnormal psychology and some type of pharmaceutical course. Due to the wide variety of people that I interact with, I think a variety of courses are important including child development courses. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? We had to complete a variety of research projects that students had to be involved in. I think those were especially helpful. Any internship in a facility would be beneficial. What advice would you give to someone who was thinking about entering the field you are in? You are not going to become independently wealthy, however, the intrinsic value of the job and know – ing that you are helping to better someone’s life is a great responsibility and reward. If you were choosing a career and occupation all over again, what (if anything) would you do differently? Probably go on to get a master’s degree in counseling. Copyright . 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career, American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. 3.1 The Basic Components of Research Designs W hen thinking about research designs, there are fundamental components or building blocks that need to be considered. The first of these is to consider whether the dependent variable is measured before and after (pretest-posttest) the introduc – tion of the independent vari- able, or just after ( posttest only). But first, let’s briefly review independent and dependent variables. The independent variable is the variable that is manipulated, controlled, or organized by the Voices from the Workplace (continued) The independent variable of an experiment is the variable that the researcher can control and change. Glow Wellness/SuperStock lan66845_03_c03_p063-100.indd 65 4/20/12 3:33 PM CHAPTER 3 Section 3.1 The Basic Components of Research Designs researcher. For example, in studying the behavioral effects of caffeine in college students, a researcher may desire to control or manipulate the consumption of caffeine during the experiment. Different students receive different amounts of caffeine, measured in milli- grams (mg). Here caffeine consumption is controlled by the experimenter, defined as the number of milligrams consumed by the student. In fact, the researcher could have various levels of caffeine consumption, such as 50 mg, 100 mg, 200 mg, and 400 mg. This type of independent variable is called a non-subject variable because the actual value of the inde – pendent variable (in this case, the number of milligrams of caffeine received) is not deter – mined by the person receiving the caffeine, but by the researcher. Sometimes the value or level of the independent variable is determined by the individual par ticipant, and this is known as a subject variable. Subject variables can only be arranged or organized by the researcher—they cannot be controlled or manipulated. For example, a person’s level of extroversion is not manipulated by the experimenter; however, it may be measured and that person assigned to a specific group (high, medium, or low extroversion). Vari – ables such as gender, personality traits, natural hair color, and race are subject variables: a characteristic each person possesses that can only be organized into different groups in a study (neither controlled nor manipulated). Just as there are different types of independent variables, there are different types of dependent variables. Remember that the dependent variable is the one that is measured— hopefully the direct result of the manipulations of the independent variable. For example, dependent variables can be either qualitative or quantitative. A qualitative variable is one in which the responses differ in kind or type. That is, there is a difference in quality (what form) rather than quantity (how many), and the outcomes of these qualitative variables are usually described in words. On a survey, if you asked someone to write a few sen – tences telling you about his or her experience today at the mall, this w ould be qualitative data. Quantitative variables differ in amount; there is more or less of some known entity. Quantitative variables are usually described by numbers, and psychologists tend to strive to develop measures of behaviors (dependent variables) that yield a number. On a survey, if you asked someone to answer multiple questions about his or her experience at the mall where 0 = terrible experience and 10 = best experience ever, this would be quantitative data. Remember that an appropriate dependent variable is the result of careful, systematic observation that is translated into a clear measure of behavior. Pretest-Posttest or Posttest Only For one of your courses you are currently taking, there may be a cumulative final exam. This is one way of thinking of a posttest only scenario. Researchers often like to use X’s and O’s to describe research designs. The X in a research design stands for some sort of intervention or independent variable manipulation—X marks the situation where some – thing is happening. The O in a research design stands for an observation or a measurement—that is the depen – dent variable. For this example, the X would be the course (X course ) and the O would be the cumulative final exam (O final exam ). We could put them in a linear sequence, as below, and this is an example of a posttest only design reading from left to right. X course Ofinal exam lan66845_03_c03_p063-100.indd 66 4/20/12 2:47 PM CHAPTER 3 Section 3.1 The Basic Components of Research Designs Notice above that there was no pretest at the beginning of the course—if there had been, that would be a pretest-posttest design, and it would look like O X O. In fact, it could be interesting to give the course final exam on the first day of class, and agai n on the last day of class. A pretest-posttest design looks like this: Ofinal exam Xcourse Ofinal exam Not to get too far ahead, but we could add a control group to this design—give the pretest and posttest to a different group of students not enrolled in the course (a control group). That design would look like this. Ofinal exam Xcourse Ofinal exam Ofinal exam Ofinal exam We’ll come back to the advantages and disadvantages of pretest-posttest and posttest only designs at the end of this chapter. For more on these designs, see Meltzoff (1998). One Independent Variable or a Factorial Design A second basic component of knowing about research designs is to know the number of independent variables being manipulated, controlled, or arranged. One independent variable is simply referred to as one independent variable, but more than one indepen- dent variable is called a factorial design. Factorial designs have some distinct advantages, namely, the ability to understand interactions between multiple independent va riables. Between or Within Groups Design Another major component of the basic building blocks of experimental design is whether the research design is a between groups design, a within groups design, or a mixture of both—in that case, a mixed design. Briefly, the between groups design is intended to mea- sure differences between separate groups of participants in a study. For example, if your col – lege or university offers a course to help students prepare for the GRE (a test often required for admittance to many types of graduate programs), and you were interested in whether freshmen, sophomores, juniors, or seniors would benefit most from the GRE course, this would be a between groups design. Four different, separate groups of individuals (fresh – men, sophomores, juniors, and seniors) were utilized to see if the GRE course was success – ful in helping students improve their GRE scores. In this case, the focus is on the difference between groups. Of course, we could have more than one between groups independent variable. We could add gender as a variable, making this a 4 (year in school) × 2 (gender) between groups design. The most common example of a within groups design would be when we are looking for a change within a participant over time (there are more complicated versions of the within groups design, and we’ll save those for later in this chapter). The pretest-posttest design (without the control group) is a good example. If you were to take a “cumulative” final exam at the beginning of the course, take the course, and then take the cumulative final exam again, this is a within groups design. The goal of that design is to see if you changed over lan66845_03_c03_p063-100.indd 67 4/20/12 2:47 PM CHAPTER 3 Section 3.1 The Basic Components of Research Designs time (that is, if taking the course led to increased scores). Of course, there are all kinds of reasons why the scores could have changed over time, and we’ll address design issues later that help us make meaningful conclusions from data. The goal of a within groups design is typically to examine how a person may change over time, whereas the goal of a between groups design is typically to examine how groups of people may differ from one another. Often the research question you are interested in dictates the type of design used. For example, if you wanted to test whether right-handed individuals have more legible hand – writing than left-handed individuals, this research design dictates a between groups design (MacKenzie, 2008) because it will take two different groups of people to make the comparison. But let’s say you wanted to know if the more you practice a new skill, the better your skill level becomes. In this case, you are looking for a change in a person over time, such as using a typing program to gain more proficiency at keyboarding skills. To detect skill development over time, within groups designs are used. But if you wanted to look at skill development over time (within groups) depending on three types of typing training programs (between groups), you could include both between groups and within groups design features into your research. This is called a mixed design. Randomization, Matching, or Blocking A final key component of experimental designs concerns how participants are assigned to certain conditions or variations of the experiment. Typically, randomization is the stron- gest or most powerful approach to assigning participants to experimental conditions. Take the case of the pretest-posttest design that includes a control group (shown again here as a reminder): O final exam Xcourse Ofinal exam (Experimental Group) Ofinal exam Ofinal exam (Control Group) The issue of random assign- ment comes into play when we have to determine which group of students composes the experi – mental group and which group of students composes the con – trol group. That process of how students are assigned to groups is very important because if true random assignment can be used, that helps to strengthen the con – clusions drawn from our data. However, this is also a good example of a case where ran – dom assignment is not typi – cally used. As researchers, we typically don’t have the power to randomly assign students to Assigning subjects to experiment condition randomly is the best approach. You can use dice or numbers to help you randomly assign subjects to your experiment. age fotostock/SuperStock lan66845_03_c03_p063-100.indd 68 4/20/12 2:47 PM CHAPTER 3 Section 3.2 Between Groups Designs course sections—students usually pick their own courses. Thus, in cases where random assignment may not be possible (Gribbons & Herman, 1997), we turn to other methods, such as matching in between groups designs and blocking in within groups designs. There are also situations where random assignment may not be an ethical choice, such as testing the effectiveness of a child welfare program and including a control group where benefits are withheld (Bawden & Sonenstein, n.d.). In an experimental design, knowing how participants are placed into the conditions of the independent variable is key—typi – cally, the strongest option is to use random assignment. Even if complete randomizatio n is not possible, we will always want to know how participants were assigned to groups or conditions. That concludes our overview of the key components of experimental designs. Now, let’s turn our attention exclusively to between groups designs, how to use them, and what we can learn with effective implementation. 3.2 Between Groups Designs I n a between groups design, the researcher is interested in comparing or detecting differences between the groups. This could be that the researcher expects males and females to behave differently, or Republicans, Democrats, and Independents to vote differently, that psychology and nursing majors have different expectations and career paths, and so on. There is no need to make this more complicated than it has to be; the goal of a between groups design is to detect if there is a significant difference between the groups being tested. To be an independent variable, the values (or levels) of that vari – able must be manipulated, controlled, or arranged (such as in subject variables). To be a between groups independent variable, there must be at least two groups, although there could be more than that. Participants can be “claimed” in only one level of the variable for it to be between groups. Using our examples from above, between groups is appropri – ate if you can only be placed in the male or female group; the Republican, Democrat, or Independent; or the psychology major or nursing major group. If you are a double major in psychology and nursing, then a between groups design would not be the appropriate design for testing. In considering a between groups factorial design, there are two types of effects that we look for: main effects and interactions. A main effect gives us information about the overall effect of each of the independent variables, whereas an interaction effect allows us to look at the combinations of the levels of the independent variables to examine if these combinations lead to different outcomes compared to other possible combina – tions. Let’s say you were doing some research on the optimum conditions for students to learn. For this example, learning is defined as a score on a test. In your research design, one between groups independent variable is how tests are administered; a test is either administered online or in a traditional classroom setting. A second independent variable in this research is how the course is delivered—either online or through a classroom set – ting. Both of these variables are between groups variables, and you will need four dif – ferent groups of participants to complete this research as envisioned. Below is how this design would look, graphically: lan66845_03_c03_p063-100.indd 69 4/20/12 2:47 PM CHAPTER 3 Section 3.2 Between Groups Designs This is known as a completely crossed design, because four conditions are being tested— two levels of test administration times two levels of course delivery. If you think about it, three of these four conditions make sense. If you were taking an online course, you could see yourself also taking your tests online. It’s not that unusual for a course taught in a classroom to sometimes use online testing, as a matter of efficiency. It would also be fairly typical for some courses that are taught in the classroom to be testing in the classroom. The last combination would be a bit unexpected; taking a course online, but being tested in a classroom. Perhaps this could happen if the instructor felt that there was a higher than normal chance of cheating that would occur if the test were given online. In any case, this is a 2 × 2 between groups design, and the way this is laid out, this design will require four separate groups of students to assess the effects of the independent variables (test admin – istration, course delivery) on the dependent variable (student test scores). A main effect examines the overall effect of one independent variable. Using the above example, will there be a main effect of test administration? That is, is there a significant difference between scores for students who are tested online versus those tested in a class- room? Essentially, this main effect looks to see if the scores in the rows in the table are different from one another. The second main effect is to look at the effect of the method of course delivery, to see if teaching the course online versus in the classroom leads to a difference in test scores. In this case, the main effect examines if the scores in the columns are different. However, what is more fascinating about this design is whether there is an interaction effect that is statistically significant. In other words, is there a combination of rows and columns (that is, a particular cell) that stands out and leads to superior student perfor – mance on tests? You can see how this information would be valuable. If the best combi – nation of student learning occurs when online instruction is followed by online testing, that would be vital information for educators to have. However, if no one combination is better than any other, that would be important too, because we would know that we were not depriving students of a learning experience that was more beneficial than another. We could have a situation where just the test administration main effect was significant, or just the course delivery main effect was significant, or different combinations, including a significant interaction. The following graphs depict these different types of outcomes. Course Delivery Test Administration Online Online Classroom Classroom This is an example of a 2 × 2 experimental design, where the rows depict the independent variable of test administration (2 levels; online and classroom) and the columns depict another independent variable, course delivery (2 levels; online and classroom). Figure 3.1: A 2 × 2 design lan66845_03_c03_p063-100.indd 70 4/20/12 2:47 PM CHAPTER 3 Section 3.2 Between Groups Designs Figure 3.2 shows an example a test administration main effect. Regardless of how the course is delivered, classroom testing is superior, leading to higher test scores. In this case, there is not a significant interaction; a significant interaction would be evidenced by a dif- ferent pattern of bars. Classroom Delivery Online Delivery Score on Test Online Testing Classroom Testing This graph depicts a main effect for the testing administration variable. That is, regardless of the mode of class delivery, student scores in classroom testing are higher than student scores using online testing. Figure 3.2: One type of main effect Classroom Delivery Online Delivery Score on Test Online Testing Classroom Testing This is a depiction of a main effect of course administration. Regardless of whether the test was administered online or in a classroom, those students receiving classroom course delivery scored higher on the test than those receiving the course via online delivery. Figure 3.3: Another type of main effect lan66845_03_c03_p063-100.indd 71 4/20/12 2:47 PM CHAPTER 3 Section 3.2 Between Groups Designs Figure 3.3 is an example of data where there is a main effect of course delivery, but no interaction. Scores on the test are higher no matter what type of testing is given, so long as the course is being delivered in a classroom. Although there are numerous examples of what an interaction might look like, Figure 3.4 shows an example of what the data would look like with an interaction taking place. There is not a simple answer to the question “who scores best” limited to the effects of only one independent variable. Here is the place where an interaction is the most meaningfully interpreted. There is one combination of test administration and course delivery that lea ds to the best combination of test scores—as you can see from the graph in Figure 3.4, that best combination is when the course is taught in the classroom but the test is adminis- tered online. This information would be highly valuable to educators and students alike. Given that test scores are measured on an interval/ratio scale, an appropriate statistical analysis would be a two-way ANOVA (two -way meaning two independent variables). The ANOVA is a statistical procedure that allows for the detection of differences when there are three or more levels of an independent variable, or two or more independent variables (with any number of levels). The interaction effect would turn out to be statis – tically significant (or not). There would be further analyses of interest. The significant interaction tells us that there is a difference between the four groups—it does not desig- nate the exact nature (or location) of the differences. For those data, post hoc analyses (or follow-up analyses) are needed. With interactions, this analysis is called simple effects, or simple main effects (Newsom, n.d.). Using a test of simple effects, we could iden- tify how the online testing/classroom delivery condition is different from the other three combinations of the independent variables. If we suspect ahead of time (i.e., a priori) that certain conditions may be different (such as online testing/online delivery versus online Classroom Delivery Online Delivery Score on Test Online Testing Classroom Testing This is an example of a classic interaction effect. Rather than test scores being higher due to type of testing (a main effect) or type of course administration (a main effect), here is an example of how two independent variables can interact. Figure 3.4: An interaction effect lan66845_03_c03_p063-100.indd 72 4/20/12 2:47 PM CHAPTER 3 Section 3.2 Between Groups Designs testing/classroom delivery), a planned comparison can be conducted without the need of the two-way ANOVA (Newsom, n.d.). Sometimes the answer to complex questions is “it depends,” and interactions allow us to figure out what is happening to our dependent variable when multiple independent variables may be influencing one anot her. Classic Studies in Psychology: Cognitive Dissonance (Festinger & Carlsmith, 1959) In 1957 Leon Festinger published an influential theory in social psychology called cognitive dissonance theory. As Festinger and Carlsmith (1959) originally characterized the theory, when a person privately holds an opinion but is pressured publicly to argue against the privately held opinion, a form of dis – comfort or dissonance will occur because of the conflict. Festinger thought of these two cognitions as not fitting together psychologically. As this theory was further studied and refined, cognitive disso- nance was also thought of as a situation where a person’s attitudes and behaviors are in conflict, and the amount of dissonance would become a predictor in the degree of motivation to resolve the disso – nance, either by changing the attitudes or changing the behaviors (Aronson, 1992). What is so interest – ing about cognitive dissonance theory is that it makes specific predictions about changes in attitudes and behaviors, and sometimes counterintuitive results occur in determining what changes attitudes (Festinger & Carlsmith, 1959). Cognitive dissonance theory provided a wealth of opportunities for future research. In the Festinger and Carlsmith (1959) study, the actual participants were students who were asked to lie to other students in a study who were about to perform a series of truly boring tasks. The participants were assigned to one of three conditions. In the control/baseline condition, the participant wasn’t asked to lie about the upcoming task to a participant waiting to complete the study. In the “one dollar” condi – tion, the actual participant was paid $1 to lie to the waiting participant, and tell him or her that the upcoming tasks were interesting, enjoyable, and fun. In the “twenty dollar” condition, the participant told the same lie as in the $1 condition, but was paid $20 to lie. In the $1 and $20 conditions, disso – nance was present—the participants knew that the tasks were dull and boring but lied about it. The participants were asked a number of questions about the study, and their responses are the depen – dent variables that Festinger and Carlsmith were most interested in. A key dependent variable question for Festinger and Carlsmith (1959) was “how enjoyable tasks were,” and the true participants (control, $1, $20) answered this on a −5 to +5 scale. The control partici- pants average was −0.45; the average for the $1 condition was +1.35; and the average for the $20 condition was −0.05. Control participants were not asked to lie to waiting participants, and when asked, they were slightly negative toward the upcoming dull and boring tasks (M = −0.45). However, the remaining two experimental conditions did lie to the waiting participants, saying that the upcom – ing tasks were interesting, enjoyable, and fun when, in fact, those tasks were indeed dull and boring. According the Festinger and Carlsmith, the participants in the $1 condition felt the most dissonance, hence they changed their own perception of the experiment (M = +1.35) to match the lie they were telling. In the $20 condition (M = −0.05), there was lesser dissonance as compared to the $1 condi – tion (M = +1.35). Why this pattern? To tell a lie for such a small amount of money, the participants had to change their own perceptions (cognitive dissonance theory). But to tell a lie for a much larger amount, participants remember it’s just a lie and take the $20 without changing their own attitudes. “The greater the reward offered (beyond what is necessary to elicit the behavior), the smaller was the effect” (Festinger & Carlsmith, 1959, p. 208). To put it another way (Aronson, 1992), “people believe lies they tell only if they are under-rewarded for telling them” (p. 304). As Aronson (1992) and others have written about, the theory of cognitive dissonance may be one of the most importance contributions of social psychology (see also Jones, 1976), and it has inspired thousands of studies. Cognitive dissonance theory was a welcome relief for many psychologists who were influenced by behaviorism at the time and the prevalence of reinforcement theory as a simple explanation for human behavior. Cognitive dissonance allowed for the study of human (continued) lan66845_03_c03_p063-100.indd 73 4/20/12 2:47 PM CHAPTER 3 Section 3.3 Participant Selection Challenges in Between Groups Designs 3.3 Participant Selection Challenges in Between Groups Designs I deally, we would select a random sample from our population of interest to be used in our research. In a truly random sample, every member of the population has an equal chance of being selected. It is not often in experimental research that we have a compre- hensive and accurate listing of every member of a population. Think about doing research in the town where you live. The roster of residents probably changes often, perhaps even daily, with people moving in, moving out, going on vacation, and so on. So researchers are often left with a nonrandom selection strategy—examples include ava ilability sam – pling (sometimes called convenience sampling), quota sampling, and snowball sampling (sometimes called chair referral sampling) (Festinger & DeMatteo, 2008). Availability sampling means that those who are selected to participate were conveniently available to behavior in a new light, in addition to the reinforcement approach, as Aronson (1992) restated Festinger’s impact in this context: “If a person held two cognitions that were psychological inconsistent, he or she would experience dissonance and would attempt to reduce dissonance much as one would attempt to reduce hunger, thirst, or any drive” (p. 304). Another important contribution of Festinger’s theory and work was to offer alternative methods for changing behaviors. Prior to this research, it was generally considered that if you wanted to change behaviors, you needed to first change a person’s attitudes—that is, our attitudes drive our behaviors (Aronson, 1992). Cognitive dissonance theory predicts that when attitudes and behaviors are in enough dissonance, behaviors may indeed change to match attitudes, but attitudes can also change to match behaviors. Aronson (1992) pointed to the convincing example of desegregation of schools in the South in the 1950s. Some psychologists sug – gested that attitudes needed to change first before changing the behavior (desegregating the schools), but cognitive dissonance theory allowed for the prediction that if you change the behaviors (integrate schools), that event can set in motion a change in attitudes, which in fact did occur (Aronson, 1992). Cognitive dissonance theory is still powerful today and has been used to analyze citizen responses to the events of September 11, 2001 (Masters, 2005). When attitudes and behaviors conflict (or simulta – neously held cognitions conflict), we are motivated to resolve the dissonance. Critical Thinking Questions 1. Can you think of situations in your own life where an attitude you publicly held was not in sync with your private behavior? According to Festinger and Carlsmith (1959), one of those two condi- tions must be resolved for the dissonance to fade. In your personal situation, which won out—did you change your attitude or your behavior? 2. Think about how cognitive dissonance might be purposely used to help attitude or behavior change. Would it be useful to point out to individuals how their attitudes and behaviors are not in sync? Thinking about your knowledge of psychology from this and other courses, what principles and theo – ries would be useful to apply to achieve an intended attitude change? An intended behavior change? 3. Think about how the idea of cognitive dissonance applies to major problems that society faces. There is a heightened awareness about global warming and environmental concerns, but look around your local parking lot and check out the types of cars being driven. Is public transportation well utilized where you live? We know about the negative effects of poverty and homelessness, but think about the efforts in your community (e.g., fund-raising, shelters). We hold certain atti – tudes and we possess knowledge, but what facilitates behavior change? Why do so many people see the problems and fail to act? How might cognitive dissonance theory explain (a) a level of relative inaction, and (b) how dissonance might be leveraged for society-level changes? Classic Studies in Psychology: Cognitive Dissonance (continued) lan66845_03_c03_p063-100.indd 74 4/20/12 2:47 PM CHAPTER 3 Section 3.3 Participant Selection Challenges in Between Groups Designs the research, such as in a subject pool for a college or university, or an online post with a URL widely distributed, and so on. Quota sampling refers to when a particular makeup of participants in the sample is desired—typically to match the makeup of the population. So if you were doing political research in a county where there the registered voters are 41% Republican, 43% Democrat, and 16% Independent, in quota sampling you wou ld want those same percentages in your sample to match the population. Snowball sampling is an informal procedure where the researcher makes an initial round of contacts to solicit participants for a study but then invites those contacts to invite others—so you might invite someone to participate in a study via Facebook, but then ask that person to invite his or her Facebook friends to partici – pate as well. In a between groups design, the overarching goal is to obtain roughly equivalent groups prior to the introduction of the indepen – dent variable manipulation. I use the phrase “roughly equivalent groups” here because although we’d prefer exactly equivalent groups, that is unlikely with chance operating and consider – ing the sheer nature of two sepa- rate groups of people being con – sidered for any type of study. Special Situations in Between Groups Designs You determined that to answer your research questions of interest (that is, to address your working hypotheses), you settled on a between groups design. With any type of design, some things should be considered when making a design decision. What follows are con – siderations with regard to between groups designs. First, demand characteristics are considered from the viewpoint of the participant. That is, sometimes experimental participants try to “figure out” the nature of the research and “help” the researcher by giving into his or her “demands” (of course, the research doesn’t demand anything, but oftentimes participants are eager to please, and researchers are eager to be pleased). To avoid demand characteristics (if possible), sometimes a research study can be designed such that the participants do not know what you (the researcher) are looking for. Recent research demonstrated that demand characteristics can influence the behavior of experimental participants (Nichols & Maner, 2008). Participants may know there is a treatment group and a control group, but the participant is unable to figure out which one he or she is in. This is called a single-blind experiment. In a single-blind experiment,participants are unaware of the experimental condition they are in. Take, for example, a study that involves weight loss as a dependent variable In a true random sample, every member of the population would have an equal chance of being selected. Is this always practical when selecting participants? Stockphoto/Thinkstock lan66845_03_c03_p063-100.indd 75 4/20/12 2:47 PM CHAPTER 3 Section 3.3 Participant Selection Challenges in Between Groups Designs measure. A researcher has designed a new over-the-counter medication to assist in weight loss. If you knew you were in the experimental group, you might be more vigi – lant about your diet, or you might take the stairs more often because you know you are receiving a drug designed to help lose weight. Although those are good ideas (eating better, exercising more), for the sake of evaluating the effectiveness of the new drug, those changes in behavior are confounds. A confound is an event or occurrence that hap- pens at the same time of your study, is not part of your designed study, but can influence the outcome of your study. For instance, if you had students in a computer lab complet – ing online surveys, and the power went out on the lab computers the stud ents were using, this is an event that occurred during your study, not part of your study, but could influence the results—students starting the study all over again when the power came back on might be frustrated and respond differently than they did originally. As for the single-blind experiment, it would be better if both groups were kept “in the dark” about their group membership until after the study was over (of course, this research would involve participants providing informed consent prior to the study beginning). In research involving psychotherapy, the demand characteristics may be the cues that communicate the therapist’s expectations, wishes, and general approach that the thera – pist uses with clients—of course these influences have the possibility of unduly influ – encing the behavior of the client during psychotherapeutic research (Kanter, Kohlen – berg, & Loftus, 2002). Case Study: Analyzing Research-Based Journal Articles It will appear at times that journal articles published in psychology are written in a foreign language. Psychology has its own jargon and methodology for presenting research findings just as so many other social sciences and sciences do. Of course you can read about how to read about journal articles, but perhaps the best practice is to just dive in and practice, practice, practice. Perhaps you have already conducted a literature search and you have some journal articles available to you. Or you have copies of journal articles you used for a previous research paper. Locate some of those articles now. To the best of your ability, try to answer the following items below, linking these questions to the answers you would extract from the journal article you have selected. The more you practice this skill, the better you will become at this skill. If you wish, you can try this on your own at first and then work with others on the task. After reading the journal article, working by yourself or in groups, think about and generate possible answers to the following questions: 1. What are the design elements and operational definitions? 2. What are the potential confounds? 3. What are the strengths and weaknesses of the study design? 4. Researchers strive to randomly assign the participants to the experimental and control conditions. What other strategies could they have used for random assignment in the study you selected? 5. What possible threats to internal validity might be created in this study? Internal validity repre- sents our confidence that the scores being measured truly represent the psychological concepts we think the scores represent. That is, the dependent variable really is what we say it is. 6. External validity represents our confidence that the conclusions from one particular study can be generalized beyond that particular study. That is, can we apply the results from the study to other situations, places, and times in history? lan66845_03_c03_p063-100.indd 76 4/20/12 2:47 PM CHAPTER 3 Section 3.3 Participant Selection Challenges in Between Groups Designs The Use of Placebos and Double-Blind Experiments In a between groups design, the single-blind experiment is designed to keep the partici – pants “in the dark” about which experimental condition they are in, so that the partici – pants do not spontaneously change their behavior due to expectancies of the research (experimental expectancy) or their own perceptions of what they are being asked for in the experiment (demand characteristics). Depending on the type of experiment being conducted, the participants may want to ask a researcher in charge of the study about the study, ask questions for clarification, and so forth. In some cases, the experiment – er ’s answers to these questions might give the participants a clue abou t the experimen – tal condition they are in. In our weight-loss study example, if the experimental group received a pill to help them lose weight, and the control group received no pill, then it would be pretty easy for the participants to determine which group they were in. In this case, the use of placebos and double-blind experimentation may be warranted. You should know that the use of a double-blind experimental procedure is not without criti- cism. For instance, both Glass (2008) and Hoffer (1999) raised issues about the use of placebo controls in research, especially in conjunction with clinical tri – als to develop new medica – tions (e.g., new medications to help in the treatment of schizo- phrenia or alcoholism). Hoffer (1999) passionately argued that double-blind experiments do not achieve the goals they set out to achieve: Do these patients know instinctively that these double-blind experiments, the gold standard of modern medicine, are perhaps best labeled as “fools gold” standard, that they are unethical, that they do not remove bias in the evaluation of treatments and that they remove the most essential element of any doctor-patient relationship, hope?” (p. 179). Of course, this refers to medical research and clinical trials, but psychologists need to be aware of such concerns when including a placebo group and double-blind study condi – tions. Much of the above discussion focuses on medical research and the clinical trial using a placebo double-blind study, but how would this design translate to an example in psychology? When I was an undergraduate, I was fortunate enough to be able to do an independent research project, studying the effects of caffeine on human performance (Landrum, Meliska, & Loke, 1988). I used a double-blind procedure in this caffeine study. Students in the experimental and control (placebo) conditions each received an 8-oz Criticisms for using placebos in double-blind experiments stem from using placebos in conjunction with clinical trials to develop new drugs. iStockphoto/Thinkstock lan66845_03_c03_p063-100.indd 77 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs serving of Diet Coke. In the experimental condition, 200 mg of caffeine was dissolved in 10 ml of distilled water, and added to the Diet Coke. For the placebo control group, just 10 ml of distilled water was added to the Diet Coke. At the conclusion of the study, we asked participants to guess whether they had received caffeine or the placebo, and a chi- square analysis indicated that participants could not reliably predict their own condi- tion—this manipulation check helps to make the case that participants were truly “blind” to the condition they were in. My undergraduate mentor, Dr. Charles Meliska of Monmouth College, prepared the 8-oz servings + 10 ml of distilled water, placed them in paper cups, and labeled the cups either “A” or “B”—he knew whether A or B contained caffeine, but I was unin – formed until after the data were collected. I was the experimenter in the room with the participants. When they asked me if they were receiving caffeine or not (of course some participants were curious—a good example of demand characteristics in play)— I honestly could not answer the question so I could not give participant s any potential clues. I (as the experimenter) was blind to conditions, and participants were blind to whether they were receiving caffeine or placebo—thus double blind. After consuming the drinks, all participants were then subjected to a number of cognitive tasks to mea – sure (the dependent variables) whether or not caffeine consumption led to changes in cognitive performance. 3.4 Within Groups Designs W hereas the first section of this chapter was devoted to a discussion of bet ween groups designs, including key features and limitations, we’ll now discuss within groups designs, along with their key features and limitations. Certain types of designs lend themselves to certain specialty areas in psychology. For example, if you were to examine the areas of learning and memory, psychophysics, and perception, you would see that many of these areas are studied utilizing within groups designs (Keren, 1993). A typical memory study would be if we had a participant memo – rize multiple lists of words—for example, a list of high-frequency words (words we com – monly hear in everyday language) and a list of low-frequency words (words not often heard in everyday language). Participants in this within groups design would learn both lists, because a key feature to a within groups design is that the participants are exposed to every level of the independent variable (word frequency), not just one level of the inde – pendent variable. Many studies in the areas of social psychology and personality rely on a between groups design. Within groups designs tend to fall into two categories: (a) the same participant is repeatedly measured on all levels of the independent variable (the participant experiences the same or similar stimuli on multiple trials), and (b) there is a comparison of scores for the same participant, but the scores come from substantially dif – ferent conditions of the experiment (Hellier, 1998; Keren, 1993). There are some key advantages in using within groups designs that are summarized here, with help from Keren (1993), Hellier (1998), and Reeves and Geiger (1994). First, there is a statistical advantage to using a within groups design. By using the same par- ticipants repeatedly in a within groups design, you increase your statistical power by lan66845_03_c03_p063-100.indd 78 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs minimizing the variability of participants. Instead of recruiting 60 individuals for a between groups design, you might only need 20 individuals, repeatedly measured, for a within groups design. This reduction in variability due to the reduction in the numbers of individuals needed increases power. The within groups design also translates into higher degrees of freedom, and with higher degrees of freedom it is easier to reject the null hypothesis. Degrees of freedom is a term from statistics that refers to the number of scores that are free to vary. Because of the design of statistical formulas in inferential statistics, adjustments are made (often such as N − 1) when calculating sample statistics, so the scores that are free to vary are reduced in number to acknowledge the amount of estimation taking place. A second advantage comes from the ability to compare participant performance across conditions. Thinking back to the previous learning and memory example, what if a third condition were added to the study—memorize a list of mixed-frequency words (some high frequency, some low frequency). If the memory task were to first memorize a list of mixed-frequency words, followed by the exclusively high- or low-frequency words, then this first “mixed” condition could serve as a baseline. Rather than have a separate control group (as you might in a between groups design), each participant would have his or her own control comparison—his or her performance on the mixed-frequency list. In this type of within groups experiment, each participant serves as his or her own control. This type of within groups comparison also helps to reduce variability and increase power, because rather than having a control group of separate individuals for comparison, each person serves as a built-in control group for him- or herself. A third advantage of the within groups design is the efficiency of the design. In the previ – ous examples, I’ve commented that fewer participants may be needed, and one still has the ability to obtain robust numbers for degrees of freedom for statistical calculations. Reducing the numbers of participants needed for an experiment makes the research pro – cess less expensive and more efficient for hypothesis testing. These are practical matters to consider when determining a program of research—whether it be doing research for your company with a bachelor ’s degree in psychology, conducting research in graduate school, or if you are psychologist. The research design decision can be complex, depending on the hypotheses to be tes ted, access to a participant pool, type of research being conducted, and so on. Furthermore, some design features can be mixed and matched so that the best features of each are uti – lized in a research design. However, the terminology can become confusing. For example, a factorial design indicates that there are at least two independent variables, but that label alone does not tell you if the independent variables are between groups variables, within groups variables, or a mixture of the two—a mixed design. Mixed Designs A mixed design also implies more than one independent variable, but this term commu – nicates a bit more information than the factorial design label. In a mixed design, there is at least one between groups independent variable and at least one within groups indepen – dent variable. There can be more than two independent variables, but we know from the mixed label that there is a mixture of between and within groups independent variables. lan66845_03_c03_p063-100.indd 79 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs You should know, however, that in some instances, the term mixed design may refer to a mixture of random- and fixed-effect variables, rather than a mixture of between and within groups variables (Maxwell & Delaney, 2004). A random-effect variable is one where a sample is drawn from a population it hopes to represent (e.g., selecting participants from a population into a sample), while a fixed-effect variable is a variable assumed to be measured without error (e.g., such as the assignment of participants to experimental con – ditions controlled by the researchers) (Newsom, 2006). Split-Plot Designs A split-plot design is a type of mixed design, and a factorial design as well. Let’s con sider the easiest example of a split-plot design—one with two independent variables. One of the variables, variable “A,” has at least two levels, and each level has a group of randomly assigned participants. The other variable, variable “B,” contains the same participants at every level of A. During 2008–2009, I worked with research assistants to design a booklet that helps students become more testwise (that is, knowing the tips and tricks for test- taking that will help you score better, even when you don’t know the answer). We devel – oped a testwiseness booklet that delivered content to the students, and we also designed a control booklet that was about college in general but did not contain testwiseness tips. Essentially, this is an experimental group (treatment group) and a control group—this is our “A” variable, and participants were randomly assigned to treatment or control. The dependent variable measurement involved the answers to general trivia questions, but within a set of 60 trivia questions, 20 items each were of easy, medium, and hard dif- ficulty (this was taken from published norms about the trivia items). The “B” variable here was item difficulty, and every participant was exposed to all three levels of difficulty (easy, medium, hard), no matter what condition of “A” he or she was in. It may be easy to think about this in picture form, so Figure 3.5 is a graphic that depicts this split plot design. Control Booklet Testwiseness Booklet A 1 Easy B 1 Medium B 2 HardB 3 Easy B 1 Medium B 2 HardB 3 A2 Between Groups Independent Variable— random assignment Within Groups Independent Variable—fixed assignment In a mixed group design, there is at least one between groups independent variable and at least one within groups independent variable; thus mixed. In this example, one group of participants received the testwiseness booklet and a separate group received a control booklet. However, all participants received easy, medium, and hard problems to solve—a within groups variable. Figure 3.5: Example of a mixed group design lan66845_03_c03_p063-100.indd 80 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs So, in a split-plot design, each level of A (A 1 and A 2 above) contains a different group of randomly assigned participants; A 1 were students who received the testwiseness booklet, and the A 2 students were a different group of students who received the control book – let. Additionally, in a split-plot design, each level of B (B 1, B 2, B 3) at any given level of A contains the same participants. For A 1, these participants receive all the levels of B (easy, medium, and hard trivia items). In the language of research designs, the whole plot factor is the variable that is applied widely, and in our example, that would be the A variable, whether or not the participant was in the experimental group or in the control group. The split-plot factor is where a variable is divided in multiple subplots, and in our exam – ple that is the B variable, or the difficulty levels of the trivia items. Connecting with our random-effect, fixed-effect discussion from earlier, in this example the A variable is the random-effect variable, and the B variable is the fixed-effect variable. Repeated Measures Designs A repeated measures design is often employed when there is some desire to moni – tor change over time. The types and varieties of repeated measures designs are fairly straightforward. One typical use of the repeated measures design is to test the same person on the same task over time. Over short intervals of time, this might be referred to as a before-and-after situation, or a pretest-posttest condition (Hadzi-Pavlovic, 1986; Price, 2000). Or, the intervals could be longer, in which case the design could be referred to as a longitudinal design. No matter the time interval, this repeated measures design focuses on the same individual responding to the same experimental situation over two or more instances. Another variety of the repeated measures design is when the same individual is exposed to multiple (yet) related levels of the same within groups independent variable (Hadzi-Pavlovic, 1986; Price, 2000). For example, this could be that in a drug study, the same person receives dosages of 5 mg/kg, 15 mg/kg, and 25 mg/kg over multiple trials in the study. In a study that I will describe in more detail later, Landrum and Clark (2006) used a repeated measures design to determine which type of chapter outline (traditional, graphical, or alpha – betical) students preferred when reading a text – book. The participants answered the same sur – vey three times, but the stimulus materials they were responding to differed in each condition. Repeated measures designs typically yield data that are correlated, because the dependent vari – able observations that repeatedly come from the A repeated measures design is used when a researcher wants to study a change over time, such as how this experimenter is studying the brain function of a participant over the course of the study. age fotostock/SuperStock lan66845_03_c03_p063-100.indd 81 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs same individual should, in fact, be related (Gueorguieva & Krystal, 2004). The correla – tion between the multiple scores in a repeated measures design is known as sphericity, and if the sphericity assumption is violated (such as an extended length of time between the repeated measures), then the statistical analysis is weakened while inflating the probability of a Type I error (Gliner, Morgan, & Harmon, 2002). In a repeated measures design, participants are more efficiently utilized because they provide data over and over again, reducing the number of participants needed. In addition, there is a statisti- cal advantage in the reduction of variability because of the need for fewer individuals, as well as the ability to compare individuals’ results to earlier data points (“each person serves as his/her own control”). These two advantages are formidable when appropriate for the experimental situation (Gliner et al., 2002). Gliner et al. (2002) also enumerated the disadvantages to a repeated measures design, such as carryover effects (mentioned later in this chapter). Basically, if in one of the conditions of the independent variable a participant has learned a new skill, for example, that skill cannot be readily “unlearned.” When looking for change, there are other concerns as well, such as ceiling and floor effects, which are addressed later. Participant Assignment: Matching or Blocking Matching and blocking represent two strategies that are used to either reduce the num – ber of explanations of how an independent variable influences a dependen t variable, or to add a finer-grained explanation about variables. Although not always the case, the matching procedure tends to be associated with a between groups design and the block – ing procedure tends to be associated with a within groups design. Part of the confusion in understanding these methodological and statistical approaches is that researchers are not always consistent in their use of terminology, and sometimes specialty disciplines use their own terminology, which may or may not be widely adopted. For example, the term “split plot” comes from agriculture research, where plots of land were split or divided into different conditions to be tested. Although we use the term split plot, you’ll sometimes see the term randomized block used synonymously for split plot (Wuensch, 2005). Matching typically refers to the pairing of participants based on similar measures on a targeted variable. Matching may allow for some statistical analysis advanta ges regarding the reduction of group variability, but “its greatest allure, however, is the hope that matching will produce equivalent groups or control for variability between groups used for matching” (Camp, 1995, p. 54). Imagine that you work in the Institutional Assessment and Research (IAR) office at your institution and you’ve been asked to conduct a study that is interested in determining a student’s satisfaction with his or her undergraduate experience. To help forge fond memories of the collegiate experience, the institution is about to require a “capstone experience” course, but for now this is just an elective, and the point of the study is to see how the capstone experience might (or might not) influence a graduate’s satisfaction level. Understanding a student’s satisfaction level could be an impor tant variable for col – leges and universities. So we set out to do a study, and of course we review the literature first, and we discover that in some studies, a student’s GPA may be related to perceptions of satisfaction. lan66845_03_c03_p063-100.indd 82 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs A student’s GPA would be considered a subject variable, because as the researcher you cannot assign someone randomly to a GPA level—it is a characteristic that students already possess. With this type of variable (a subject variable that we suspect is related to our dependent variable of interest—student satisfaction), we would consider using a matching or blocking strategy. Let’s consider matching first. You might retrieve the cur – rent GPAs from the registrar ’s office of all of this year ’s graduating seniors. In the follow – ing example, 20 seniors make up the sample. To see the benefit of matching, let’s consider the worst-case scenario first. If you retrieved those 20 GPAs from the registrar ’s office and rank ordered them from high to low, they might look like what is depicted in Figure 3.6. From these 20 students, you ran- domly assign 10 to the “experi – mental” condition, which would be to take the capstone experi – ence course, and 10 to the control condition, who do not take the course. Your comparison will be between the experimental condi- tion and the control condition on the dependent variable, col- lege satisfaction. But if you know from the review of the literature that GPA is a key variable that influences college satisfaction, you might want to control for GPA. One method of controlling for it would be to have roughly equivalent groups prior to the introduction of the independent variable manipulation (the cap – stone course). But say, in this worst case scenario, that you left group assignment to chance, and by some fluke of randomization, the top 10 students in the list were assigned to the experimen – tal group, and the bottom 10 students were randomly assigned to the control group (See Figure 3.6; the solid line separates the top 10 from the bottom 10). These two groups would not be roughly equivalent on GPA before introducing the capstone experience independent variable. In fact, the mean for the top 10 students is 3.68 (SD = 0.2), and the mean for the bottom 10 students is 3.00 (SD = 0.2). Sometimes leaving group assign – ment to randomization alone may not be good enough (especially with a small N). One approach to help achieve at least roughly equivalent groups would be to use matching, as depicted in Figure 3.7. 4.0 3.9 3.9 3.8 3.7 3.7 3.6 3.4 3.4 3.4 3.3 3.2 3.1 3.1 3.0 3.0 3.0 2.9 2.8 2.6 In a sample of 20 students, this could be a distribution of GPAs that have been arranged from high to low. Even by using randomization, groups can be non-equivalent prior to the start of a study. Figure 3.6: A hypothetical distribution of GPAs lan66845_03_c03_p063-100.indd 83 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs In this rank-ordered list of GPAs, the nearest pairs have been matched (as indicated by the ovals), and then from within each oval, there has been ran- dom assignment into either the experimental condition or the control condition. This procedure won’t guarantee equivalent GPAs for the two groups prior to the introduction of the independent variable, but the groups should be much closer. In fact, the mean GPA for the experimental group is now 3.36 (SD = 0.4), and the mean GPA for the control group is 3.32 ( SD = 0.4). The benefit of match- ing is that this procedure roughly equates the groups on the extra – neous variable (GPA) before the study begins. Thus, after the study is over, if the capstone experience does indeed positively influence student satisfaction, we can say with some confidence that GPA is not the driving force behind the improvements in student satisfac – tion, because GPA was roughly equivalent in each group prior to the capstone experience. By using matching, we attempt to wipe out any possible influence of GPA. Blocking , however, takes a differ – ent approach. Rather than wipe out the effect of GPA, by using blocking, GPA is turned into a variable of interest in the experiment, which allows us to examine potential interactions between GPA and how the capstone experience might influence student satisfaction. With blocking, we turn the potentially extraneous variable (GPA in our example) into an independent variable, which will allow us to examine if this variable interacts with the intended independent variable, capstone expe rience. Perhaps the capstone experience course is only beneficial for influencing studen t satisfaction with high GPA students. If that is the case, then the blocking design will capture that interac – tion, whereas in the matching design, the effect of GPA is erased by matching the pairs and then randomly assigning to experimental and control conditions. So in keeping with our example, we would start by “blocking on GPA,” as depicted in Figure 3.8. –experimental 3.6 –experimental 3.4 –experimental 3.3 –experimental 3.1 –experimental 3.0 –experimental 2.9 –experimental 2.8 –experimental 3.7 –experimental 3.9 –experimental 3.9 –control 4.0 –control 3.8 –control 3.7 –control 3.4 –control 3.4 –control 3.2 –control 3.1 –control 3.0 –control 3.0 –control 2.6 This is a graphic depiction of matching, which may be used to avoid a situation where randomization alone may not ensure roughly equivalent groups. After arranging the participants from high to low on a GPA measure, matched pairs are identified (the circled pairs), and then a coin flip is used to determine which participant is assigned to the experimental group and which participant is assigned to the control group. Figure 3.7: An example of matching lan66845_03_c03_p063-100.indd 84 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs In this example, we have purposely grouped or blocked the higher GPA students together and the lower GPA students together, because we are going to “block on GPA” and turn GPA into an independent variable. In blocking, an individual block needs to be composed of a homogeneous (similar) group of individuals—in this case, relatively high GPAs. Thus, this new independent variable has two GPA levels—high and low. In the next step of blocking, each group of homogeneous participants is split so that the effect of the capstone experience independent variable can be estimated— see how this is depicted in Figure 3.9. 3.6Higher GPA Group Lower GPA Group 3.4 3.7 3.9 3.9 4.0 3.8 3.7 3.4 3.4 3.3 3.1 3.0 2.9 2.8 3.2 3.1 3.0 3.0 2.6 In blocking, a group of individuals is “blocked” together as a group (as compared to matched in a matching design). Eventually, the block will be split into experimental and control groups; sometimes this is called a split-plot design. Figure 3.8: An example of blocking lan66845_03_c03_p063-100.indd 85 4/20/12 2:47 PM CHAPTER 3 Section 3.4 Within Groups Designs Thus, with the randomized block design , participants were grouped into blocks based on their GPAs, then each block was split randomly, and then each split plot was randomly assigned either to the experimental group (capstone experience) or control group. In this way, GPA becomes a variable in the experiment, and now we can test to see if GPA inter – acts with the capstone experience in affecting student satisfaction. Both the matching and blocking design offer advantages and disadvantages. For the matching design, the effect of a potential extraneous/nuisance variable can be washed away, and a potential confound can be minimized. However, the matching procedure adds a great deal of effort to the experimental procedure. All of the matching must be completed prior to the actual study commencing. With GPAs from the registrar ’s office, this might not be so much work, but if you were to match on variables that require more effort (such as self-esteem levels or intelligence), this would add to the time and resources necessary for matching—and researchers are able to match on more than one variable, so the complexity of matching can increase dramatically. The advantage of blocking is that it adds the ability to ascertain the influence of the variable in question (GPA) into the experimental mix, and rather than wiping it out (as in the matching approach), blocking allows us to observe potential interactions. However, blocking involves more work, and it is not always easy to find homogeneous groups of participants that can be adequately split. In sum, both techniques can be useful additions to your methodolo gical arsenal, but it is important to know what each can do for you, as well as be aware of the additional costs in terms of time, resources, and energy expenditure. 3.6 Higher GPA Group Higher GPA Split Plot Higher GPA Split Plot Lower GPA Split Plot Lower GPA Split Plot Experimental Group Control Group Experimental Group Control Group Lower GPA Group 3.4 3.7 3.9 3.9 4.0 3.8 3.7 3.4 3.4 3.3 3.1 3.0 2.9 2.8 3.2 3.1 3.0 3.0 2.6 After forming groups of homogeneous participants on a variable of interest (in this example, GPA), the high GPA group can be split into two plots, and then one receives the independent variable manipulation and the other serves as the control group. This procedure is again repeated for the low GPA group; split into plots and then experimental conditions are assigned. Figure 3.9: An example of a split-plot design lan66845_03_c03_p063-100.indd 86 4/20/12 2:47 PM CHAPTER 3 Section 3.5 Special Issues Using Within Groups Designs 3.5 Special Issues Using Within Groups Designs A s with any type of design approach, there are advantages and disadvantages, and some of these ideas have been alluded to in previous parts of this chapter. When utilizing a within groups design, there are a number of issues to consider, but many of these issues have strategies for resolution. In addition, occasionally there are design limitations that may hinder the interpretation of an outcome (or point to an alterna – tive approach that may potentially yield superior results). We’ll address some of the major issues facing within groups designs, such as accounting for pretreatment differences and ceiling and floor effects, and dealing with order effects through counterbalancing. Floor and Ceiling Effects In within groups designs that are particularly focused on the analysis of change over time, such as in a repeated measures design, the consideration of potential floor and ceiling effects is important. Floor effects and ceiling effects are related to the notion of regression toward the mean (Altermatt, 2008). A floor effect occurs when you are working with scores at the very low end of the distribution of scores. If you start with individuals with very low scores, and you give them the same test again (as you would in a repeated measures design), those at the low end cannot go any lower than low, and some of the scores might increase by default. In essence, very low scores have nowhere else to go but up. The problem with this floor effect is that you may see an increase in scores and think that your independent vari – able is effective, when the increase in scores is due to something else: regression toward the mean . Think about the distribution here. The example that Altermatt (2008) uses to indicate the floor effect is when giving a group of second graders a sixth-grade spelling test. Second graders would typically not do so well on such a test. But after the first administration of the spelling test, you take all the children who scored a 0 (at the very bottom of the distribu – tion), give them the same spelling test again, and some might get a cou ple of words right, just by guessing or having a second exposure to the words. Thus, scores could increase, and the average score would creep toward the mean. The floor effect occurs when starting with scores so low that there is an inability for those scores to go any lower than low. The floor effect occurs when you start with low scores, and the goal is to change scores and make them lower. The starting location of the scores (low) has an impact on the ability to measure change, even if the intervention was believed to be effective. Figure 3.10: The floor effect lan66845_03_c03_p063-100.indd 87 4/20/12 2:47 PM CHAPTER 3 Section 3.5 Special Issues Using Within Groups Designs As you can imagine, there is a similar process at the other end of the distribution, which is called a ceiling effect. Using Altermatt’s (2008) example, you would give a group of sixth graders a second grade spelling test and then take those tests with the highest scores and re-administer the second grade spelling test. Those students scoring at the high end can either retain the same high score, or score lower, which would be a regression toward the mean. The decrease in scores is merely an artifact of starting so high (i.e., the ceiling effect), and the scores have no other direction to go than down. Ceiling and floor effects are a concern in within groups designs, especially repeated mea – sures designs, because of the emphasis of detecting a meaningful change ove r time. But ceiling and floor effects limit meaningful change interpretations because of the regression toward the mean factor. So what would be the better solution? One alternative would be to include a control group and see if the rates of change over time are similar or different when comparing an experimental group to a control group. Another solution would be to try to avoid selecting individuals from the extremes of the distribution—if scores have the ability to move in either direction (up or down), then ceiling and floor effects can be tested, and the researcher has a better chance of detecting meaningful change if present. The ceiling effect occurs when you start with high scores, and the goal is to change scores and make them higher. The starting location of the scores (high) has an impact on the ability to measure change, even if the intervention were believed to be effective. Figure 3.11: The ceiling effect lan66845_03_c03_p063-100.indd 88 4/20/12 2:47 PM CHAPTER 3 Section 3.5 Special Issues Using Within Groups Designs In an attempt to demonstrate that an intervention is effective, it is preferred to select participants neither at the floor nor at the ceiling, such as those highlighted in the figure, and attempt to show change. It is a fairer test of a hypothesis to be able to support or refute it; thus, avoiding potential floor and ceiling effects is preferred. Figure 3.12: Avoiding floor and ceiling effects Carryover Effects, Order Effects, and Counterbalancing When using a within groups variable, each participant is exposed to every level of that independent variable. Having a participant complete multiple tasks can have some impli – cations as to how the completion of earlier tasks can influence the completion of later tasks. This general concern is typically labeled as a carryover effect—that is, the effect of one level of the independent variable can persist to influence another level of the inde – pendent variable (Goodlet, 2001). Carryover effects can lead to progressive error, which means that factors other than the independent variable are influencing the dependent variable over time. Two basic carryover effects are practice and fatigue (Goodlet, 2001; Hall, 1998). When performance on an earlier trial in the experiment positively influences later results (due to practice, experience, or familiarity), this is known as a practice effect, or positive progressive error . When an earlier trial negatively influences later results (due to fatigue, boredom, or inattention), this is known as a fatigue effect, or negative progres- sive error (Goodlet, 2001; Hall, 1998). Let’s consider an example of a practice effect (positive progressive error)—hopefully this hasn’t happened to you too many times. A student will sometimes try very hard but may eventually fail a class. Often a student will retake the class with the same instruc – tor, because even though the student failed, he or she is familiar with the instructor ’s lecture style, method of testing, classroom interactions, and so on. Think of this situation as a repeated measures design, where the first trial was the first time the student com – pleted the course and the second trial is the second time through the same course with the same instructor. When evaluating how much the student learned from the second time through the course, it is difficult to know this with certainty, because there is a carryover effect from the first time through the course. Certainly the student benefited from already knowing the instructor ’s style of teaching, as well as having taken all the instructor ’s tests before (even if the actual questions did change from semester to semester). Of course, in lan66845_03_c03_p063-100.indd 89 4/20/12 2:47 PM CHAPTER 3 Section 3.5 Special Issues Using Within Groups Designs this example, a practice effect is beneficial to the student, but from a researcher ’s perspec- tive, this positive progressive error complicated our interpretation of just how much the student learned in the second completion of the course. Repeated trials can also lead to fatigue effects, or negative progressive error. For example, you and your classmates might each develop a short survey in your Resear ch Methods class, and ask a group of students to complete your surveys so that you can collect real data and apply statistical analyses to your data. If each student prepares a 10-item survey, and there are 20 students in your course, then you will be asking participants to complete 200 survey items. Think about that task—will the attention level of your participants be the same on survey item No. 10 as it is on survey item No. 190?. Participants might get fatigued by answering so many survey questions, or just get bored, and their attention may wander. So if your questions are at the end of survey, you may have less confidence that the participants answered your questions in a serious manner, which ultimately impacts the conclusions drawn. So how do we minimize the impact of carryover effects? First, you should note that in all situations, we may not be able to reduce carryover effects. For instance, when a student takes a course and then retakes a course, since this is not an experiment, we lack control over the sequence of events. However, in our survey scenario, we could vary the order of presentation of the surveys such that the survey questions for Student 20 do not always occur last in the sequence, and we could order (or rearrange) the sequence so that Student 20 is not as concerned about fatigue effects. The technique that we use to minimize poten – tial carryover effects is called counterbalancing. Although there are various approaches to counterbalancing, we will focus here on a general approach known as across-subjects counterbalancing (Goodlet, 2001), where dif – ferent participants in a within groups design receive different orders of the levels of the independent variable. The goal of counterbalancing is to vary the or ders to such a degree than any order effects are dissipated, or balanced out. For example, for 20 stu – dents asking a 10-item survey each, Student 20 does not always have his or her questions appear last, but different students take turns with their survey questions occurring last. “The basic idea of counterbalancing is to spread any order effects evenly across experi – mental conditions so that order effects will not be confounded with experimental treat – ments” (Wuensch, 2007, ¶5). In complete counterbalancing, every possible combination of orders is presented to different groups of participants. As briefly mentioned earlier, I did research looking at the types of outlines students prefer with their textbooks (Lan – drum & Clark, 2006). In that study, we tested three types of outlines: (a) a traditional, Roman-numeral outline; (b) a graphical outline, using a concept map to connect related concepts; and (c) an alphabetical listing of the key terms from the chapter, not presented chronologically. At the end of each outline presentation, participants answered questions about their preference and use of the outline presented. This study used a counterbalanc- ing strategy known as complete counterbalancing, because we presented every possible order. With 3 levels of the within groups variable (traditional, graphical, alphabetical), there are 6 possible orders—to calculate the number of possible orders, use N!, where N is the number of levels (3), and the ! (factorial) symbol means that you multiply that whole number by every whole number less than that down to the number 1— such as in 3*2*1 = 6 orders. You can see the orders in Table 3.1. lan66845_03_c03_p063-100.indd 90 4/20/12 2:47 PM CHAPTER 3 Section 3.6 Limitations of Experiments Table 3.1: Outlines and possible orders Order 1TraditionalGraphicalAlphabetical Order 2 TraditionalAlphabeticalGraphical Order 3 GraphicalTraditionalAlphabetical Order 4 GraphicalAlphabeticalTraditional Order 5 AlphabeticalTraditionalGraphical Order 6 AlphabeticalGraphicalTraditional In the Landrum and Clark (2006) study, at least 24 participants completed each of the six orders, and we used an “order” independent variable to see if the order of presentation influenced the evaluation of the usefulness of the outlines. Sometimes, we discovered, order did matter! Complete counterbalancing can be useful with small numbe rs of levels of the within groups independent variable, but the numbers accelerate quickly. If you had 4 levels of the independent variable, it would take 24 different orders (4!). If you had 5 levels of the independent variable, it would take 120 different orders (5!). And, going back to our 20 students enrolled, if you wanted to present every possible order of 20 different surveys to participants, it would take 2.43 × 10 18 different orders—not a task we would attempt. So when the total number of orderings is not possible or realistic, we utilize par – tial or incomplete counterbalancing approaches (Goodlet, 2001; Wuensch, 2007). Within groups designs bring advantages as well as particular challenges, such as carry- over and order effects. Even with these creative workarounds, there are occasions when a within groups design is just not feasible. This occurs when one level of the with in groups independent variable may lead to irreversible changes, such as learning (Hall, 1998). For example, what if you were to do a study on the different types of educational approaches used to teach someone how to use the latest version of Microsoft Word? You might have three different approaches, such as a paper-based tutorial, live classroom instruction, and a series of podcasts with instructions. You could pay attention to order effects and vary the order of presentation of the levels of the variable, but can you see how whatever l evel comes first may irreversibly change the participant? If you were to present a series of instructional podcasts about how to use the newest version of Microsoft Word, once you completed that training, you cannot undo what you have learned and start fresh with the paper-based tuto – rials—you are no longer a blank slate, and the learning from the first condition may carry over into the second condition. Sometimes, even with all the methodological controls that we use, a within groups design may not be the best choice, depending on your hypotheses of interest, and you may have to consider other designs, such as a between groups design. 3.6 Limitations of Experiments E xperiments, like those presented throughout this textbook, provide the strongest ability to draw cause-and-effect conclusions from research. This type of conclusion is very powerful; if we understand cause and effect, we can implement interven- tions and strategies to help positive events occur more often, as well as work to minimize lan66845_03_c03_p063-100.indd 91 4/20/12 2:47 PM CHAPTER 3 Section 3.6 Limitations of Experiments negative events. In the following chapters, we’ll address research scenarios where experi- mentation is not typically possible—quasi-experimental designs, single-subject designs, and surveys and questionnaires. In these latter situations, we can learn a great deal about human behavior, but the strength of our conclusions is typically not as strong as those conclusions drawn from experiments. However, as you will quickly see, not all situa – tions allow experimentation, nor random assignment to conditions, nor st rict experimen – tal control of the situation, and so forth. Thus, we have multiple tools in the t oolbox to address those situations. Even though the experiment can yield valuable information, it too has li mitations, as does any approach. In many ways, the experimental method is revered for elegance of its logic and the strength of conclusions drawn from it, but it is not the one-size-fits-all solution for research in psychology. Those psychologists who perform work in the field or clini – cal studies are often unable to utilize the strict experimental controls of the laboratory (Levine, 1974). One of the limitations of the experiment is “that there is no such thing as a social vacuum” (Levine, 1974, p. 663). Just because a participant is brought into a laboratory setting does not mean that the participant is not influenced by the social context of the setting, the experimenters, the task, and so on. Levine expands on this theme when he reminds us that important human problems involve whole human events that occur in a his – torical and social context. What does this imply? Even though you may be conducting an experi – ment in a laboratory setting using adequate methodological controls, it does not mean that behavior observed in the laboratory is represen – tative of the actual behavior that would be seen outside of the laboratory. Two other related concerns offered by Levine (1974) include that (a) researchers become a part of the phenomenon they are researching, and we are influenced by the processes we use, and (b) science is essentially a social enterprise, no matter to what extent we strive for objectivity. To summarize, what do these ideas mean in the midst of your applied project? These are the fun – damental ideas to keep in mind: 1. No method, not even an experiment, will yield the type of certain results that most scientists desire. 2. Some scientists overstate the importance of experiments and may not be a ware of the social context that influences all that we do. 3. Many research situations exist (e.g., clinical work, fieldwork) that are not amena – ble to experiment/laboratory experimentation as we know it, yet these research situations have scientific merit. Clinical and field studies are limited in their ability to control conditions. What is meant by the phrase “there is no such thing as a social vacuum”? Photodisc/Thinkstock lan66845_03_c03_p063-100.indd 92 4/20/12 2:47 PM CHAPTER 3 Chapter Summary 4. Although experiments may yield studies with high internal validity, the general – izability of the research (i.e., external validity) may be limited. 5. Researchers need an arsenal of tools (in addition to methodologies used in experi – ments) to study and understand attitudes and behaviors in the real world. To achieve this last goal, the chapters that follow will explore quasi-experimental designs, where we apply the fundamental principles of experiments (as best we can) to partici – pants, topics, and scenarios that do not lend themselves well to experim entation. In another chapter you’ll learn about single-subject research, where much can be gained from the systematic study of one individual. A common research technique that will also be explored is the use of surveys and questionnaires, and how to adequately prepare such instruments to allow for meaningful interpretations. Rather than seeking a one-size- fits-all solution to research scenarios, the goal is to provide you with a versatile collection of tools so that you can apply appropriate methodological tools to appropriate research questions. Chapter Summary T he basic building blocks for any experimental research include (a) whether or not the dependent variable is measured before and after the independent variable manipulation, or just after; (b) whether there is one independent variable or more than one (more than one independent variable receiving the label “factorial design”); (c) whether the primary question of interest is the difference between different groups of people (between groups design) or the change in the same people over time (within groups design); and (d) the primary method by which people are assigned to the condition of an experiment (e.g., matching or blocking). Although the effect of one independent variable can be instructive (a main effect), the complexity of human behavior is more likely to be captured by interactions; that is, when the levels of one independent variable combine with the levels of another independent variable, leading to a unique dep endent variable measure. The potential deficiencies in a between groups design are often addressed by the strategies used to assign participants to conditions, such as double-blind studies to avoid demand characteristics and the placebo effect. Within groups research primarily addresses the potential changes in a person over time, such as before-and-after measure (called pretest-posttest). Different types of within groups designs (split plot, repeated mea – sures) are utilized to answer different types of research questions. Each design presents its own unique challenges, and ceiling, floor, carryover, and order effects are just some of the design issues that are addressed in within groups research. Ultimately, an experiment approach can be quite helpful in answering certain types of questions, but o ne size does not fit all. Other research approaches are necessary to study questions of interest where random assignment is not possible and/or is unethical. lan66845_03_c03_p063-100.indd 93 4/20/12 2:47 PM CHAPTER 3 Concept Check Research designs from this chapter Type of StudyDesign Name Symbolic Representation X = treatment or intervention O = observation, dependent variable measurement Brief Features Experimental or Quasi-Experimental Pretest-Posttest O X O Allows for measure of change over time. Pre-measure helpful to ensure equivalent groups. However, pre-measure also sensitizes participants to later measures. Experimental or Quasi-Experimental Posttest Only X O Measures effect of treatment without sensitization. However, pre-existing group differences are not detectable without pretest. Experimental or Quasi-Experimental Pretest-Posttest Control Group O X O O O Enjoys the benefit of change over time, but control group adds a strong comparison. If time alone accounts for the change, the control group pretest-posttest will capture that effect. Concept Check 1. Twenty-seven patients at a clinic with stage three lung cancer are randomly assigned to traditional medical treatment, homeopathic treatment, or a combina – tion of the two. Based on the literature, researchers predict that the combination will be most effective in reducing the size and number of existing tumors. In this study, what is the independent variable? A. 27 patients at a clinic B. Lung cancer patients C. Size and number of existing tumors D. Medical, homeopathic, or combined treatment 2. Which of the following would most likely be a non-subject variable? A. Test scores B. Personality C. Age D. Dental history lan66845_03_c03_p063-100.indd 94 4/20/12 2:47 PM CHAPTER 3 Questions for Critical Thinking 3. Placebos A. cannot be used in research involving social sciences. B. are people who act like participants but are in on the experiment. C. allow study conditions to be conducted as double-blind. D. are not permitted for human subjects research. 4. An advantage of matching participants is that it A. decreases the number of variables needed. B. reduces the group variability. C. increases the sample size. D. highlights individual differences. 5. One method to control for floor and ceiling effects is to A. use multiple independent variables. B. include a control group. C. use a pretest-posttest design. D. measure only one dependent variable. Answers 1. D. Medical, homeopathic, or combined treatment. The answer can be found in Section 3.1. 2. A. Test scores. The answer can be found in Section 3.1. 3. C. Allow study conditions to be conducted as double-blind. The answer can be found in Section 3.3. 4. B. Reduces the group variability. The answer can be found in Section 3.4. 5. B. Include a control group. The answer can be found in Section 3.5. Questions for Critical Thinking 1. Think about an aspect of psychology that you are extremely curious about. What type of between groups study would be useful in providing answers to your hypotheses? What would be the independent variables in your study, and what would be the dependent variables? Practice answering these key design ques – tions—would you ask questions pretest-posttest or posttest only? How would individuals be assigned to your levels of the independent variable? 2. Now consider the same scenario for a research question that would utilize a within groups approach. Is there a topic where you would be interested in change over time? How would you account for potential ceiling and floor effects? Is the topic you selected to study with a within groups approach one that could be converted to a between groups design or does it only work for a within groups design? 3. Experiments are valuable in that they can often lead to a cause-and-effect conclu – sion about how the levels of the independent variable bring about change s in the dependent variable measures. Can you generate some examples of when an experiment would be inappropriate? Brainstorm about other possible approaches to gaining information about these types of research scenarios, and then keep reading the next few chapters of this textbook for ideas on how to study phenom – ena that do not lend themselves to experimentation. lan66845_03_c03_p063-100.indd 95 4/20/12 2:47 PM CHAPTER 3 Key Terms to Remember Key Terms to Remember ANOVA ANalysis Of VAriance A statisti- cal procedure that allows for the detection of differences when there are three or more levels of an independent variable, or two or more independent variables. availability sampling When the individu- als selected to participate were conve- niently available to the research. between groups design A method of study design is that intended to measure differences between separate groups of participants in a study. Ex: freshman, sophomores, juniors, and seniors. blocking A process of data analysis that turns a potentially extraneous variable into an independent variable, which permits the examination of whether or not the vari- able interacts with the intended indepen- dent variable. carryover effect The idea that the effect of one level of the independent variable can persist to influence another level of the independent variable. ceiling effect When a test does not have the ability to identify performance accurately because of the lack of difficult test items. cognitive dissonance theory A theory developed by Festinger and Carlsmith that occurs when a person privately holds an opinion but is pressured publicly to argue against the privately held opinion, and a form of discomfort or dissonance occurs because of the conflict. counterbalancing A technique that is used to minimize potential carryover effects in an experiment. degrees of freedom A statistical term that refers to the number of scores that are free to vary. demand characteristics When experimen- tal participants try to figure out the nature of the research and “help” the researcher by giving into the perceived demands. dependent variable The variable that is measured. double-blind experimentation When neither the study participants nor the experimenter are aware of the conditions being administered during the course of an experiment in order to prevent bias. factorial design The statistical experiment design in which more than one inde- pendent variable is being manipulated, controlled, or arranged. This enables the experimenter to understand interactions between multiple independent variables. fatigue effect When an earlier trial negatively influences later results due to fatigue, boredom, or inattention. See nega- tive progressive error. fixed-effect variable A variable assumed to be measured without error. floor effect Occurs when you are work – ing with scores at the very low end of the distribution of scores that do not have the potential to go any lower. Thus, sub- sequent attempts yielding improvement may not be accurate because of the scores’ inability to decrease. homogeneous Variables or conditions that are similar in nature. lan66845_03_c03_p063-100.indd 96 4/20/12 2:47 PM CHAPTER 3 Key Terms to Remember independent variable The variable that is manipulated, controlled, or arranged/ organized by the researcher. interactions An effect that allows us to look at the combinations of the levels of the independent variables to examine if these combinations lead to different outcomes compared to other possible combinations. main effects The overall effect of each of the independent variables considered individually. matching The pairing of participants based on similar measures on a targeted variable. mixed design When an experimenter includes both between groups and within groups design features into his or her research. negative progressive error When an ear – lier trial negatively influences later results due to fatigue, boredom, or inattention. See fatigue effect. non-subject variable When the value of the independent variable is not determined by the participant but rather by the researcher. planned comparison When an experi- menter decides which comparisons to conduct when the experiment is being designed and develops questions relevant to that comparison. positive progressive error When perfor – mance on an earlier trial in the experiment positively influences later results due to practice, experience, or familiarity. See practice effect. post hoc analyses Analyzing the data after the experiment has been conducted to find patterns that were not outlined in the experiment development. posttest only When the independent vari- able is measured only after the experimen- tal intervention has been administered. practice effect When performance on an earlier trial in the experiment positively influences later results due to practice, experience, or familiarity. See positive pro- gressive error. pretest-posttest When the independent variable is measured both before and after the experimental intervention has been administered. progressive error When factors other than the independent variable are influencing the dependent variable over time. qualitative variable A variable in which responses differ in kind or type. The outcomes of these variables are usually described in words. quantitative variable A variable for which there is some known entity. The outcomes of these variables are usually described in numbers. quota sampling When a particular makeup of participants in the sample is desired because of its similarity to the gen- eral population. random assignment When participants are randomly assigned to a group or condition in an attempt to control for any significant differences among groups. random-effect variable The condition where a sample is drawn from a popula- tion it hopes to represent, such as select- ing participants from a population into a sample. randomization When individuals are assigned to a study group by chance and not in a predictable manner. lan66845_03_c03_p063-100.indd 97 4/20/12 2:47 PM CHAPTER 3 Web Resources Web Resources Pretest-posttest design and explanation of different scenarios in which this experimental design is appropriate to use. Additionally, the challenges associated with this experimen- tal design are also outlined. http://www.experiment-resources.com/pretest-posttest-designs.html Publication regarding the social psychology of the psychological experiment. It outlines a research study that examined the type of social interaction that psychological research causes and analyzes why these interactions are significant. http://www.psych.upenn.edu/history/orne/orne1962amerpsychol776783.html randomized block design When partici- pants are grouped into blocks based on a determined variable and then the blocks are randomly split for assignment to an experimental or control group. regression toward the mean When an experimenter sees a change in scores and thinks the independent variable is effec – tive, but this change in scores is due to something else. repeated measures design When the experimenter wishes to examine change over time by administering a condition to the same participant over a period of time. roughly equivalent groups Obtaining groups that are as similar to one another as possible through randomization or another technique because of the unlikelihood of obtaining exactly equivalent groups. simple effects A statistical test that seeks to identify how the one condition is different from alternative combinations of other inde – pendent variables in the same experiment. single-blind experiment When the par – ticipant is unaware of the experimental condition he or she is in. snowball sampling The informal proce- dure where the researcher makes an initial round of contacts to solicit participants for a study, but then invites those contacts to invite others to participate. sphericity The correlation between the multiple scores in a repeated measures design. split plot When a variable is divided in multiple subplots. split-plot design A type of mixed design, and a factorial design where experimen- tal conditions are grouped, such as study guide versus no study guide, and are then separately compared to different sub- groups, such as easy questions, medium questions, and hard questions, to accu- rately determine effectiveness. subject variable A characteristic, such as GPA, that an experimenter cannot randomly assign because the participant already has that characteristic before par – ticipating in the study. within groups design An experiment design that aims to measure the change within a participant over time. lan66845_03_c03_p063-100.indd 98 4/20/12 2:47 PM CHAPTER 3 Web Resources A sample chapter regarding experimental research and variable manipulation with a strong emphasis on test design and the different types of designs a researcher could choose to use. http://www.southalabama.edu/coe/bset/johnson/lectures/lec9.htm This website details 2×2 factorial design by giving examples and visual representation of different design development strategies. htt p://web.mst.edu/~psy world/mixed_desig ns.ht m lan66845_03_c03_p063-100.indd 99 4/20/12 2:47 PM lan66845_03_c03_p063-100.indd 100 4/20/12 2:47 PM 4 Quasi-Experimental Designs Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • understand the important role that quasi-experimental designs play in answering research questions of interest when participants in a research study cannot be randomly assigned to conditions. • articulate the appropriate research scenarios for a number of quasi-experimental design approaches, including nonequivalent control groups, time series, cohort/panel, and regres – sion discontinuity designs, as well as the applications of program evaluation. • appreciate the practical applications and limitations of observational designs such as case studies and naturalistic observation. • understand the contexts in which archival research approaches, such as content analysis and meta-analysis, can answer interesting questions based on the availability of pre-existing data. Associated Press lan66845_04_c04_p101-130.indd 101 4/20/12 2:47 PM CHAPTER 4 Introduction Introduction I n the ideal world, psychology researchers would like to be able to understand cause- and-effect relationships as they pertain to attitudes and behaviors. Cause-and-effect conclusions are powerful, because they provide insight as to how positive behaviors can be promoted and how negative behaviors can be lessened. Not all research situations lend themselves to laboratory controls and randomization of participants to conditions. So outside of the laboratory (and even sometimes in the laboratory), we apply research methods as best we can, but without random assignment. Much of the research conducted in the social sciences is conducted using quasi-experimental designs. In fact, there are times where random assignment is inappropriate, such as withholding health care ser – vices from children for the sake of an experimental study (Bawden & Sonenstein, n.d.). Say that I wanted to study how effective the use of clickers (student response systems) might be in the classroom. I’m teaching two sections of introductory psychology, and I decide that I will use clickers in one of the sections, and I won’t use clickers in the other section. At the end of the semester, I want to see if students earned more points in the sec – tion with clickers than in the section without clickers. This would be a quasi-experimental design, because I am unable to randomly assign students to sections (students usually like selecting their own classes). In fact, using X and O, here’s what the design would look like (remember, X represents the independent variable manipulation and O represents the dependent variable measure): X clickers O points O points If I had been able to randomly assign participants to conditions, there would have been an R (for randomization) at the beginning of each line. This design is called a nonequivalent control groups posttest only design. Although it is a good idea to have a control group, because of the lack of randomization, we are less confident that the groups were equivalent even before the start of the study—thus, we label the design as nonequivalent. This is one example of the types of designs that are collectively known as quasi-experimental designs. In this chapter, we’ll investigate the advantages and disadvantages of quasi-experimental designs, observational designs, and archival research. Voices from the Workplace Your name: James K. Your age: 47 Your gender: Male Your primary job title: Co-Director of Community Living Your current employer: Village Northwest Unlimited How long have you been employed in your present position? 27 years (continued) lan66845_04_c04_p101-130.indd 102 4/20/12 2:47 PM CHAPTER 4 Introduction What year did you graduate with your bachelor’s degree in psychology? 1982 Describe your major job duties and responsibilities. I am responsible for coordinating services for nearly 60 clients. Our clients are all adults with physi- cal or mental disabilities or both. I also help oversee the 60 staff that provides care and services for these individuals, including meeting with them on a quarterly basis to review the care programs being provided, conducting performance reviews. I help develop new training goals, and provide guidance to staff and clients on how to effectively complete their work and grow. I conduct Quality Assurance on documentation samples that are written by our staff, and perform follow-up reports to the various staff teams that we supervise. One of our goals is to encourage and support our staff, and we make this a priority in our work. What elements of your undergraduate training in psychology do you use in your work? I would say the training we received in counseling and helping techniques is probably the most benefi – cial and frequently used education from my undergraduate training. What do you like most about your job? The flexibility of schedule. What do you like least about your job? Bureaucracy and poor communication with and from the various funding streams that we work with and those hired to oversee this. Poor wages for staff. What is the compensation package for an entry-level position in your occupation? It would be a range of $19,000–$25,000 for a middle management person, $15,000–$18,000 for a full- time direct care staff. What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? Major Medical Health Insurance, Life Insurance, various perks offered by our agency such as reduced costs for various items like newspaper subscriptions, wellness or fitness memberships. What are the key skills necessary for you to succeed in your career? Patience, good listening abilities, creativity understanding, integrity, honesty, and willingness to go the extra mile. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? Abnormal Psychology, Theories of Counseling, Developmental Psychology. As an undergraduate, do you wish you had done anything differently? If so, what? I don’t think so, I was adequately prepared in my education for my job. What advice would you give to someone who was thinking about entering the field you are in? Be focused and dedicated to the people you are serving or trying to help. They are someone else’s son or daughter and have tremendous value. Try to instill ways of making your job FUN and enjoyable. Encourage and support those that work with you. Create a climate of participative decision-making so that everyone feels invested in what is being decided. Show honor and respect to those above you and to those you come into contact with each day, make them feel valued and important. Voices from the Workplace (continued) (continued) lan66845_04_c04_p101-130.indd 103 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types 4.1 Quasi-Experimental Design Types A quasi-experiment is “a design that manipulates the presumed case and measures the presumed outcome but does not randomly assign participants to conditions” (Shadish & Cook, 2009, p. 619). The example with the clickers is a nonequiva – lent control groups design, but sometimes there isn’t even a control group (these types of designs are sometimes referred to as pre-experimental designs; Morgan, Gliner, & Har – mon, 2000). Using our X’s and O’s, here is an example: X O The above is a posttest only design (commonly used) but yields very little information that can be generalized beyond the participants being tested. A common example would be the teaching evaluations you provide at the end of the semester. The instructor taught the class (X), and you provided an evaluation (O) at the end of the semester. Although helpful to the instructor, there is not much we can say about whether this instructor is effective as compared to other instructors, if one class learned better than another class, and so on. Another pre-experimental design is the one group pretest-posttest design: O1 X O2 In this case, we can observe if there was change from pre to post, but little else. Staying in class, perhaps at the beginning of a math course you are given a comprehensive pretest (O 1), and at the end of the course (X) you are given the same test again (O 2). This design will allow us to detect change over time, but that may not provide much useful informa – tion to the instructor. What if you were a math wizard coming into the course, and over the semester you didn’t learn much? That might make it look like the instructor didn’t do a very good job, when it was your excellent preparation that explains why your scores didn’t increase much over time. Note that in discussions about K–12 teacher salaries and merit- based pay, the ability to demonstrate student learning over time is a huge issue. So this pre-experimental design does allow us to inquire about change over time, but it provides little insight as to why changes may have occurred. We could also add a pretest to this non – equivalent control groups design, making it a pretest-posttest design, as depicted below: O X O O O If you were choosing a career and occupation all over again, what (if anything) would you do differently? I can’t think of anything that I would choose to do differently. Copyright . 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. Voices from the Workplace (continued) lan66845_04_c04_p101-130.indd 104 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types Although random assignment is not achieved with this design, the existence of a pretest does allow us to explore whether the groups were different at the start on the variables that were measured. This is not the same as randomization, but at least if the groups were roughly equivalent prior to the introduction of the independent variable, then our confi – dence increases as to what the possible implications of the results mean. Nonequivalent Control Groups The nonequivalent control groups design is quite common throughout the social sci – ences and psychology, and here we’ll discuss just a sampling of practical applications of this design. For example, to measure the impact of a computer-based training program for nurses, Hart et al. (2008) administered a pretest questionnaire, delivered information about evidence-based practice, and then administered a posttest questionnaire. This is the classic pretest-posttest design, and here’s what it would look like graphically: Oevidence-based practice pretest Xcomputer based education program Oevidence-based practice posttest This type of design lets the researchers know if the participants changed over time. How – ever, it’s hard to gauge the effectiveness of the intervention (X) without a control group. Sometimes the constraints of the situation make random assignment impractical. For example, a medical school decided to implement a new form of ethics training for its stu – dents based on small-group ethics teaching (Goldie, Schwartz, McConnachie, & Morrison, 2001) but wanted to compare this new approach with the previous lecture-style large- group ethics instruction. Rather than randomly assign students to different instructional conditions, new incoming medical students received the new curriculum, and students from the previous year were utilized as the control condition. The experimental design would look like this: Osurvey score Xnew curriculum Osurvey score (experimental group) Osurvey score Xold curriculum Osurvey score (control group) Luckily, under the old curriculum, an ethics and health care survey had been adminis – tered both pretest and posttest. These same instruments were utilized with the new small group ethics discussion sections. Goldie et al. (2001) found that the new curriculum led to greater consensus in considering ethical situations and concluded that “small-group ethics teaching, in an integrated medical curriculum, had a positive impact on first-year students’ potential ethical behavior. It was more effective than a lecture and a large-group seminar-based course in developing students’ normative identification with the profes – sion of medicine” (p. 295). Even though a true experiment was not conducted here, you can see the benefit of the outcomes of the quasi-experimental design—we can learn much from these types of designs, even if we cannot draw a cause-and-effect conclusion. Time Series Design In its simplest form, quasi-experimental research using a time series design “. . . is sim – ply a set of repeated observations of a variable on some entity or unit, where the num – ber of repetitions is relatively large” (Mark, Reichardt, & Sanna, 2000, p. 353). For exam – ple (Garson, 2008), the monthly calculation of the national unemployment index by the lan66845_04_c04_p101-130.indd 105 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types Bureau of Labor Statistics would be considered a simple time series design. In essence, you can think of this as an extended sequence of dependent variable measurements (O). A simple time series would look like this: O O O O O O O O O O O O The above 12 observations could be the monthly reporting of the unemployment index, for example. As you can imagine, the time series design allows for the assessment of change over time—trends—but it can do much more than that (Mark et al., 2000). A time series design can also be used for forecasting. For example, if an economist is tracking unem – ployment rates, he or she may use this data to try to predict what will happen six months from now, based on the data accumulated leading up to this point in time. Time series designs can become more complex as we introduce independent variables (X) in to the mix, such as a particular treatment or intervention. These types of designs are sometimes called interrupted time series designs (Cook & Campbell, 1979; Mark et al., 2000) because of the interruption (X) over the series of observations (O’s). This type of design might look like this: O O O O O O X O O O O O O Note the independent variable manipulation in the middle of the sequence of observa – tions. This interrupted time series design is often used to measure the impact of legisla – tion and public policy, such as the implementation of a man – datory seat belt law or a ban on cigarette smoking on a col – lege campus. Let’s say you were interested in determining the impact of a smoking ban on your college campus. You might utilize a research design where observers are trained to collect data at various points on cam – pus during different days of the week at different times of day. The dependent variable of inter – est (O) is the number of smokers observed. You implement this data collection program well before the announcement of the new smoking ban on campus that begins on July 1. Here’s what that design might look like: OJan OFeb OMar OApr OMay OJun XJul OAug OSep OOct ONov ODec OJan Notice that there are six observations before and after the ban takes effect. The goal of the smoking ban would be to reduce the number of smokers, at least the number of smoking incidents observed on campus. One of the benefits of this type of design is that it tracks If you wanted to study the impact of a smoking ban on a college campus, you could collect data by observing the number of people smoking throughout the day. iStockphoto/Thinkstock lan66845_04_c04_p101-130.indd 106 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types changes over time. So after the ban is publicized and implemented July 1, there might be an immediate decrease in observed smokers (maybe they stopped smoking, or are hid – ing it better). But after a while, say a couple of months, the numbers of observed smokers might increase. (Ever received a speeding ticket? Did you decrease your speeds for the short term, only to return to your regular habits a short time later?) So you can see the ben – efit of the interrupted time series design, to assess the impact of an intervention. But the drawback of the quasi-experimental design is that we cannot be overly confident about causality—a decrease in observed smokers could mean many things—some stopped smoking, some hid their smoking better, some switched to chewing tobacco, and so on. Time series designs can be expanded to examine the relationship between multiple levels of an independent variable manipulation (Mark et al., 2000). This design might look like this: O O O O O O X O O O O O O O O O O O O O O O O O O In fact, this is very similar to the non-equivalent control groups pretest-posttest design, but it includes multiple pretests and multiple posttests. An example of this type of design is the work of Ivancevich (1974) looking at the impact of a management-by-objectives (MBO) approach as well as reinforcement schedules in the performance of a manufactur – ing corporation with multiple plant sites. Ivancevich measured multiple dependent vari – ables, such as the quantity of output, quality of output, grievance rates by employees, and absenteeism. Three different plants were utilized, as indicated below. Plant 1 O XMBO O O O O Plant 2 O XMBO O O O XReinf. O Plant 3 O O O O O From this research, Ivancevich concluded that the benefit of the reinforcement observed in Plant 2 tended to overshadow the MBO effects. In considering time series designs, there are three factors to keep in mind (Garson, 2008a): age, period, and cohort. In the time series design, variables are repeatedly measured over time. Given that people can naturally change over time without any exposure to an inde – pendent variable, sometimes it will be difficult to disentangle a change due to the indepen – dent variable, or just the passage of time. There are also period effects as well, meaning that individuals from a particular historical period may be impacted similarly. Garson suggested that those individuals who lived through the Great Depression, the Challenger explosion, or 9/11 may have similar beliefs and behaviors. Thus, recording times series data over long periods of time (e.g., decades), the researcher must understand that historical events may be impacting the change or lack of change seen in a time series design. Finally, cohort effects should be considered. A cohort effect occurs when people from a particular age range are impacted differentially by a historical event. For instance, Garson (2008) suggested that after World War II, young adults at this time (a cohort) reacted differently than past generations to the challenges of the later Vietnam War. Thus, the idea of a cohort effect is the intersection of a group of people in time with a historical event that impacts that group; the post–World War II cohort appeared to be more suspicious of the U.S. government and its policies as com – pared to earlier cohorts. Some quasi-experimental studies are specifically designed to exam – ine cohort effects, and these types of designs are called cohort designs and panel studies. lan66845_04_c04_p101-130.indd 107 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types Cohort Designs and Panel Studies As Alwin, McCammon, and Hofer (2006) pointed out, cohorts can take on a number of identities. Sometimes researchers think of cohorts as generations, such as baby boomers or millennials. The term birth cohort is used to identify a group of people who were all born in the same year. But a general definition of cohort is a group of people who have shared critical experiences over the same span of time; however, a cohort may not be equivalent to a generation, nor may a collection of adjacent cohorts necessarily be considered a gen – eration (Alwin et al., 2006). In thinking about the three keys to time series (age, period, cohort), we can construct a standard cohort table that depicts these factors simultaneously (Alwin et al., 2006; Garson, 2008). In the cohort study, individuals are randomly sampled for each of the “cells,” thus a measure of net change over time (period), age, and cohort can be acquired (see Figure 4.1). As you can see from above, age is depicted across the rows, period is depicted down the columns, and the cohorts are identified by the diagonal lines crossing individual cells. For just a moment, think about the complexity of this design. First, your study takes 30 years to complete (launched in 1980, completed in 2010). Second, you recruit many people from many age ranges, from 20-year-olds to 89-year-olds. In the typical cohort study, partici – pant selection is random; that is, different people participate at the different periods of time. To follow a cohort, you would follow the diagonal line as time passes and participants age. You should know that not all quasi-experimental cohort designs involve 10-year time peri – ods where the study spans 30 years. Many, if not most, cohort designs are based on shorter Year of Study (Period) Age 2010 2000 1990 1980 20–29 30–39 40–49 50–59 60–69 70–79 80–89 This figure depicts the complexity of a standard cohort approach over time. In 1980, there are random samples of individuals from each of the age groupings on the leftmost column. Ten years later, additional random samples are taken for each of the age groupings depicted in the leftmost column. Note that this study would take 30 years to complete. Source: Wuench (2003) Figure 4.1: Standard cohort approach for a longitudinal study lan66845_04_c04_p101-130.indd 108 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types periods. In fact, relatively short (6 months to 1 year) cohort studies have been used to examine the effectiveness of new therapeutic approaches to help couples (Epstein et al., 2007), minor depressive disorders, and miscarriage (Klier, Geller, & Neugebauer, 2000) and to monitor the development of social skills in kindergartners (Hall, Jones, & Claxton, 2008). Panel studies , also called prospective cohort studies, add an interesting twist to the stan – dard cohort study—they follow the same individuals over time. In a typical cohort study, like the one depicted previously, different individuals are recruited and studied in 1980, 1990, 2000, and 2010. In the panel study/prospective cohort study, using this same struc – tural example, the same people would be tracked for 30 years, which would take enormous effort, expense, and organization. These types of studies are relatively rare, yet particularly informative. To illustrate a panel study, the chart has been modified to add start points and arrows to indicate that the same individuals are being studied over time (see Figure 4.2). An impressive example of this type of study was conducted by Wilson and Widom (2008), who studied individuals who had been victims of abuse and neglect during childhood. These researchers conducted a 30-year follow up with the original participants using a prospective cohort design. Although the results are complex, the major finding was that individuals who were maltreated as children were more likely to report sexual contact before age 15, participate in prostitution by their young adult years, and test positive for HIV during middle adulthood as compared to individuals who were not maltreated as children. These types of studies are invaluable in helping psychologists understand the role of childhood experiences and how these experiences may shape choices and behav – iors in various stages of adulthood. Year of Study (Period) Age2010 2000 1990 1980 20–29 30–39 40–49 50–59 60–69 70–79 80–89 This figure depicts the complexity of a panel study approach over time. In 1980, there are samples of individuals from each of the age groupings on the leftmost column selected. These same individuals are tracked over time and tested 10, 20, and 30 years later. Thus one age grouping is followed over time. Note that this study would also take 30 years to complete, but is much more complicated than a standard cohort table design because in the panel study the same individuals are tracked and tested over time. Source: Wuench (2003) Figure 4.2: Panel study or prospective cohort study for longitudinal research lan66845_04_c04_p101-130.indd 109 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types Regression Discontinuity Designs The regression discontinuity design (RDD) is a unique member of the quasi-experimental design family in that it leads to conclusions that are similar in strength to randomized controlled trials (Lesik, 2006; Rutter, 2007). Regression discontinuity designs are some – times called cutoff-based designs because assignment into the treatment group or the control group is based on a predetermined cutoff (Shadish & Cook, 2009). From a popu – lation of individuals, a cutoff score is set before the study begins, and then the trends (regression lines) are followed to see if they are continuous or not. Think about it this way—an RDD is a good choice if you are working in a situation where some remedia – tion may be helpful to a subgroup of the population (Lesik, 2006), such as students who need extra help with math, or employees who may be facing layoffs in a weak economy. Rather than randomly assigning participants to conditions, such as with a coin toss, the RDD sets a cutoff score, and scores above the cutoff are assigned to one condition of the study, while scores below the cutoff are assigned to the other condition of the study (Lesik, 2006). Cutoff Pretest 2 8 10 12 14 18 Post- test 20 22 3 T T T TT T T T TT TTT T T T T C C C C C C C C C CCCC C C C CC C C 456789 10 11 This graphic depicts the first part of a regression discontinuity design, where the relationship between two variables (here: pretest and posttest) is identified. Now, an intervention is about to be applied to the T (treatment) individuals. Source: Lesik (2006) Figure 4.3: The first part of a regression discontinuity design lan66845_04_c04_p101-130.indd 110 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types An example with pictures might be helpful here (Wuensch, 2003). Look at the graph in Figure 4.3. On the x-axis is the pretest, and on the y-axis is the posttest score. The T’s and C’s represent individual people, graphically depicting their placement with regard to their pretest and posttest scores. The solid blue line is the regression line—this is the line of best fit, the line that best captures the linear relationship between the pretest and post – test scores. As you can see, there is no “discontinuity”—the line is solid and continuous. The vertical black line just below the pretest score of 6 represents the cutoff. Thus every person scoring just below a 6 is placed into the treatment group (depicted by T’s), and everyone scoring above the cutoff is a control participant (C’s). If, after the study was complete, the results looked like this graph, this would tell us that there was no effect of the intervention/independent variable on the treatment group—in other words, the regression line shows continuity. But what if there had been an effect? What would that look like? Well, as in most research, we are looking for the independent variable to influence the dependent variable. In RDD we have that evidence when we see a regression discontinuity—a break in line (and/ or different line slopes) on either side of the cutoff. See the example in Figure 4.4 from Wuensch (2003). CutoffPretest 2 0 10 12 14 18 Post- test 20 22 T T T T T TT T T TT TT T T T CC C C CC CC C CC CCC C C CC CC C C 46 81012 16 This is a hypothetical example of a regression discontinuity design where the intervention (the treatment, T) was effective. Rather than a linear (straight-line) relationship between the pretest and the posttest, the lines are now broken (i.e., discontinuous). Source: Wuensch (2003) Figure 4.4: The second part of a regression discontinuity design lan66845_04_c04_p101-130.indd 111 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types The dashed line to the left of the cutoff reminds us about the original continuous regres – sion line, and the break in the solid lines on either side of the cutoff demonstrates a regression discontinuity. In this case, the treatment worked—posttest scores are higher (the new red line) for the lowest-scoring pretest participants as compared to where we would have expected them to be. You can see where RDD could be very useful when scores are used to determine membership in programs or receipt of benefits, such as need-based programs for scholarships, or scores that are used as cutoffs to determine which children are enrolled in gifted and talented programs in elementary schools. One key consideration to remember is that the cutoff score must be determined prior to the beginning of the study (Lesik, 2006; Rutter, 2007; Shadish & Cook, 2009). RDD adds another useful approach to studying behavior outside the laboratory in the use of quasi- experimental designs. Program Evaluation Quasi-experimental approaches are often employed in program evaluation efforts because of the emphasis on evaluating a program in its context, not in the laboratory. The basic idea of program evaluation is to assess the impact of a program based on its pre-stated goals (Rose & Fiore, 1999). But what do we mean by a program? A program tends to serve a known population, is long-lasting, and attempts to solve a limited set of prob – lems (Greene, 2003). Examples of programs that would be suitable for program evaluation would be a local Head Start program, the McNair Scholarship program, or perhaps the ongoing performance of your Department of Psychology. Greene (2003) provided excel – lent examples of what program evaluation is not : it is not a single methodology or data collection instrument, not data collection that occurs at the end of a program cycle, and not decisions made about the value or worth of a program. Program evaluation efforts tend to follow in three distinct phases, as outlined by Greene (2003). In the generation phase , a needs assessment may be conducted prior to the imple – mentation of a program, to see if the need justifies the implementation of a program, followed by program development and refinement. In the implementation phase, the program begins, provides services to clients and the community, and hopefully has a positive impact. In the causation phase , the program evaluator attempts to demonstrate and measure the impact of the program, such as the delivery of services to clients, clients’ evaluation of services, and the impact felt by other agencies or the community in general. For example, a legislator might be concerned about reports of unplanned teenage preg – nancy in his or her legislative district. Following a program evaluation approach, the first step would be to generate possible programs that could address the concern of unwanted teenage pregnancies, using local or national programs that promote abstinence, contra – ception, or both strategies. The needs analysis conducted prior to program implementa – tion would provide baseline data as to the true nature and scope of the issue—is the teen pregnancy rate on the rise, or are there just some well-publicized recent events that bring this topic to light? Following a needs analysis, a public relations (PR) campaign may be implemented to try to change teenagers’ attitudes and opinions about sexual activity, with the ultimate goal to influence behavior. This might be implemented, for example, by a state agency such as a Health and Welfare District, which might provide workshops for middle school and lan66845_04_c04_p101-130.indd 112 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types high school sex educators to offer curriculum material for teaching teenagers strategies to avoid unwanted pregnancy. Throughout the program, evaluations may be conducted, asking local sex education teachers about their perception of how the program is work – ing. Additionally, teens might be surveyed about program perceptions and any possible behavior changes. In addition, the program evaluator might examine epidemiological data from the county or legislative district to examine if the rate of teen pregnancy did decline over the period of the implementation of the program. Although it continues to be difficult to draw causality conclusions when using a quasi-experimental design (Rose & Fiore, 1999), this convergence of data might be used by citizens and legislators alike to evaluate the effectiveness of the program and determine whether the program should continue. In fact, program evaluation and other quasi-experimental approaches are often used to help answer public policy questions, as well as to generate new ideas on how to solve complex social issues. Case Study: Answering Public Policy Questions Although naturally implied, public policy questions should hopefully lead to public policy answers, and this is just one of the domains of program evaluation and the use of experimental and quasi- experimental designs. The need for increased program evaluation services comes, in part, with the necessity for agencies to be held accountable for government-sponsored human services programs. According to Hosie (1994), “program evaluation plays an extremely important role in policy-making. Outcome data is [sic] provided to decision makers who determine the value and worth of programs based on outcome and available financial resources” (p. 349). McCartney and Weiss (2007) took this notion (at least) one step further when they stated “policies designed to improve the life chances of children and families often live or die on the basis of the findings from evaluation research” (p. 59). It seems clear that program evaluations are used to affect federal, state, and local agency policy deci – sions (Hosie, 1994). Given this weight, it should be obvious how important it is to get appropriate answers in the course of addressing public policy questions. Historically, psychologists have had some success in helping to answer public policy questions, but as you’ll see, psychology can do much more. Lodzinski, Motomura, and Schneider (2005) suggested that the best public policy influence in psychology was the role psychological science played in 1954 in influencing the Brown v. Board of Education Supreme Court ruling, which essentially made racial segregation in public schools unconstitutional. The contributions of psychologists, especially Kenneth B. Clark, made the argument that segregating White and African-American students led to poorer self-esteem for African-American students and that segregation was a source of interracial prejudice. In the U.S. Supreme Court ruling, psychological research was openly credited with influencing the deci – sion, and this case has been credited with forever connecting psychology and public policy (Lodzinski et al., 2005). Critical Thinking Questions 1. Quasi-experimental designs are complex yet allow for research conclusions to be drawn outside of the laboratory. Thinking about your current or future work environment, can you think of one real-world situation where you could use a nonequivalent control groups design, times series design, cohort design or panel study, and regression discontinuity design? 2. The research methods presented in this chapter provide you with impressive additions to your toolkit or Swiss Army knife regarding how research methods can allow for the study of chal – lenging, real-world questions. However, good science communicates outcomes, especially if the goal is to influence public policy. How would you present one of the methodologies (continued) lan66845_04_c04_p101-130.indd 113 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types There are some fundamental lessons that evaluators and public policy makers can keep in mind when examining the effectiveness of a program evaluation (McCartney & Weiss, 2007): • Use mixed method designs—in other words, use different approaches to attempt to capture the totality of the program. If true experiments (e.g., randomized controlled trials) are not possible, then use quasi-experimental methods to help measure the effectiveness of programs and public policy implementations. • Interpret effect size in a research context—when conducting statistical analyses on the data from program evaluation research, one must go beyond the typical inferential/null hypothesis significance testing and look at effect sizes. This helps researchers distinguish between statistically significant artifacts and conditions where the results indicate practical significance. • Synthesize all research findings—this chapter ends with a presentation of meta- analysis, which is a methodological procedure that allows researchers to combine the results of many studies into broad recommendations and findings. • Adopt fair and reasonable scientific expectations—scientific data may go only so far in helping to answer public policy questions. Researchers must be careful not to overstate the relative impact or contribution of a set of data. • Encourage peer and public critique of the data—we need to invite our peers and the public to scrutinize our work and the resulting impact that our work has in impacting the research community. Ultimately, the success that psychologists have in influencing public policy decisions may heavily rest on the public’s abil – ity to interpret what we do. In the day-to-day functioning of a scientist, data are paramount, but policy makers and the public may not similarly appreciate a data-driven approach. McCartney and Weiss (2007) reminded us that in addition to empirical data, policy makers and citizens consider anecdotal evidence and testimonies, newspaper features, politics as usual, and so forth. Add to this mix the influence of blogs and cable news programs that feature partisan approaches, and there are many voices that contribute to the complexity of decisions to be made. Public policies, whether we like it or not, are political entities, and the success or failure of public policy can lead to the success or failure of those in government. Psy – chology needs to inform the public on what it can provide to public policy table, for the potential contributions are formidable. In 2002, Philip Zimbardo, president of the Ameri – can Psychological Association, made this statement not long after the tragic events of 9/11 regarding relevance of psychology to the shaping of public policy: discussed so far, say regression discontinuity design, to the general public? That is, how would you translate the benefits of this design approach to actual data such that others can see the benefit and comprehend the impact? Note: This is a valuable skill—if you want to see if you understand a concept, attempt to teach that concept to someone else. 3. Large social issues, such as public schooling, unemployment, teen pregnancy, universal health care, poverty, and homelessness, are complex on many levels. How might different aspects of research methods (that is, different features in your ever-growing Swiss Army knife of available tools) be used to address such large, complex societal concerns? Case Study: Answering Public Policy Questions (continued) lan66845_04_c04_p101-130.indd 114 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types Classic Studies in Psychology: Work Decisions and Absenteeism (Lawler & Hackman, 1969) If you had the chance to develop a program at work that could add to your paycheck, would you take that opportunity, or would you be skeptical and cynical about what management was up to? If the plan were designed by coworkers at another location but imposed on you, would you gladly accept the opportunity to earn more, or would you not invest your efforts into the program, since you had no hand in designing it? These are some of the real questions that Edward Lawler and Richard Hack – man began to answer in 1969. These researchers worked with a large service that cleaned buildings in the evenings, and there were vari – ous groups of employees who did not work in similar locations but performed similar tasks. Essentially, there were three groups in this quasi-experimental design. It should be noted that Lawler and Hackman (1969) purposely chose this design because conduct – ing the study in the field would increase the probabil – ity that the results of this research could be general – ized to other settings (i.e., enhance external validity). The first group in the study is labeled the “participa – tive” group, and these workers designed their own pay-for-performance incentive plan, with the help of researchers and the supervision of manage – ment. The second group in the study is the “imposed” group, which received the exact same incen – tive plan as the participative group, although the imposed group had no say in its development; thus, it was imposed upon them. The third group was the control group, and in actuality, there were two different versions of control group. One control group talked to researchers about the pay for perfor – mance plans, absenteeism, and turnover, but no changes were made to its incentive plan. Another control group received no information at all about the study and was not contacted by researchers or top management about the study. As you can see in Figure 4.5, the researchers looked at the percentage of scheduled hours worked, both before the intervention (pay plans) and after. The graph presents the data for the participative group. For example, if an employee were scheduled to work 40 hours in a week but was only pres – ent for 32 hours, on the y-axis this would yield a score of 80%. When averaged together, employees in the participative group worked 88% of their scheduled hours before the incentive plan was intro – duced. After the incentive plan, they worked 94% of their scheduled hours (a statistically significant increase). Flirt/SuperStock (continued) Even before the September 11 th tragedy, most of the major problems facing the United States were psychological in cause, correlates, or consequence: for example, AIDS and sexually transmitted diseases; drug addiction, as well as addictions to smoking, gambling, alcohol, and food; prejudice and discrimination; delinquency; violent crimes; educational failures for too many minority youth; and the full range of physical illnesses that are influ – enced by lifestyle and behavioral functioning. Psychologists have much to say about more effective ways of dealing with these problems at both individual and community levels of action. Psychologists need to be heard and to be at the table of influential leaders and policymakers because psy – chologists have more to say about these issues than do members of any other discipline (p. 432). lan66845_04_c04_p101-130.indd 115 4/20/12 2:47 PM CHAPTER 4 Section 4.1 Quasi-Experimental Design Types Now consider the results of the “imposed” group, displayed in Figure 4.6. This group received the same incentive plan, with the only difference being that its members did not design it or contribute to its creation. The data for the imposed group are below. Before the incentive plan was introduced, employees in the imposed group worked 83% of their scheduled hours; after the incentive plan, they also worked 83% of their scheduled hours—no change. Both control groups saw no change either from pre–incentive plan to post–incentive plan. Weeks Before After Percent of Scheduled Hours Actually Worked 12 12 13 14 15 16 11 11 10 10 9988 7 7 66 55443322 11 100 95 90 85 80 75 70 Participative group results from Lawler and Hackman study. Source: Lawler and Hackman (1969) Figure 4.5: An example of quasi-experimental data Weeks Before After Percent of Scheduled Hours Actually Worked 12 12 13 14 15 16 11 11 10 10 9988 7 7 66 5544332211 100 95 90 85 80 75 70 “Imposed” group results from the Lawler and Hackman study. Source: Lawler and Hackman (1969) Figure 4.6: Another example of quasi-experimental data Classic Studies in Psychology: Work Decisions and Absenteeism (Lawler & Hackman, 1969) (continued) (continued) lan66845_04_c04_p101-130.indd 116 4/20/12 2:47 PM CHAPTER 4 Section 4.2 Observational Designs Because this study was so well designed, it leads to some fascinating conclusions. Lawler and Hack- man (1969) found that “the data show that employee attendance improved only in those groups that participatively developed their own incentive plans. Neither the incentive plan alone nor participation and discussion alone yielded any changes in attendance” (p. 470). Clever design, including the con – trol groups, leads to this meaningful conclusion. If the incentive plan were all that was needed, then the imposed group would have seen increases too. If just discussion were enough, one of the control groups would have shown increased in the percentages of hours worked. The conclusion here was that participation and implementation of the plan were necessary to see significant increases in work; either option alone was ineffective. This study has been influential in informing the literature about how top management teams operate (e.g., West & Anderson, 1996), as well as in educating us on when pay-for-performance works, and when it may not (Beer & Cannon, 2004). Critical Thinking Questions 1. It is one thing to talk about abstract ideas in the context of a textbook, but when people start to discuss personal issues like salaries, you can imagine there might be some concern when researchers want to enter a situation and implement programs that could affect an employee’s paycheck. What role do researchers have in educating the public (and educating research partici – pants) about the potential beneficial role (as well as potential drawbacks) to experimentation? 2. Completing coursework in a 5-week session requires diligence, motivation, and determination. Think about the courses you have completed in this mold, and the different design features of those courses. Were there conditions or scenarios in some classes that motivated you to work harder than in other classes? Was that due to the content of the course, the design of the course, or both? Using a quasi-experimental approach, how might a researcher study how a course is structured and its relationship to student learning? 3. Can you look at a graph, study it, and decipher its meaning, or do you just gloss over graphs and figures? Are you more convinced by a persuasive story when in is presented in words (text) or pictures (graphs)? Why is that? What steps do you think you would need to take to become better versed in your non-preferred modality? Classic Studies in Psychology: Work Decisions and Absenteeism (Lawler & Hackman, 1969) (continued) 4.2 Observational Designs W ithin the scope of this chapter, a thorough and comprehensive review of obser – vational research is just not possible. We’ll focus here on two key approaches— case studies and naturalistic observation, with a brief overview of the termi – nology most relevant to this topic (Brown, n.d.; Garson, 2008; Pope & Mays, 2006). Some designs are summarized in Table 4.1. lan66845_04_c04_p101-130.indd 117 4/20/12 2:47 PM CHAPTER 4 Section 4.2 Observational Designs Table 4.1: Terminology TypeBrief Description Field experiments A field experiment involves a research study where the actual data collection occurs in natural settings (in the field). Case study An extensive observation of an individual or a single group is the hallmark of the case study approach. Case studies tend to look at a limited set of behaviors rather than the totality of the person or group. Naturalistic observation Using naturalistic observation, the researcher is involved in the direct observation of behavior as it occurs in its natural setting. In principle, the researcher does not interact within the environment being observed, but only observes. Participant observation In participant observation, the researcher inserts him- or herself into the environment being studied, which can be especially useful when studying group processes. Researchers using this technique must be careful to remain objective and avoid observer effects (those who know they are being observed may change their own behavior due to the observation). Action research Action research is a subset of participant observation in which the researcher in the natural environment works to change some aspect of behavior or the organization. These actions are designed to improve conditions for the participants or the organization. Rather than test a hypothesis, action research attempts to overtly change behavior. Archival research Archival researchers study the records that already exist that were originally recorded in natural settings. Surveys Surveys are a versatile methodological approach because they can be administered to individuals in natural settings as part of a fieldwork approach. Program evaluation Program evaluation involves the evaluation of systematic programs in applied settings. That is, program effectiveness is determined by how patients or clients are served in the field. Ethnography Ethnography involves the direct observation of people during daily life. This term is sometimes used interchangeably with case study, and ethnography refers to both a research process and the type of report that is written as a product of that research. Case Studies The case study approach focuses on a particular case of interest, and this case may be a person, a group, or perhaps an organization. Case studies can utilize qualitative and quan – titative methods. In fact, a research strategy called triangulation encourages researchers to study the variable of interest from multiple perspectives, and not over-rely on any one research approach. Researchers using a case study approach can be forward-looking (pro – spective) or look back in time (retrospective); they can approach theories inductively or deductively; and they can strive to describe, evaluate, or explain behavior (Garson, 2008b; Walsche, Caress, Chew-Graham, & Todd, 2004). Because the approach is so diverse, it is lan66845_04_c04_p101-130.indd 118 4/20/12 2:47 PM CHAPTER 4 Section 4.2 Observational Designs often difficult to define the case study, but the overriding concern may be the case of inter- est, rather than the methodological approach used (Stake, 1994): Case study is not a methodological choice, but a choice of object to be stud – ied. We choose to study the case. We could study it in many ways. The physician studies the child because the child is ill. The child’s symptoms are both qualitative and quantitative. The physician’s record is more quan – titative than qualitative. The social worker studies the child because the child is neglected. The symptoms of neglect are both qualitative and quan – titative. The formal record the social worker keeps is more qualitative than quantitative. In many professional and practical fields, cases are studied and recorded. As a form of research, case study is defined by interest in individual cases, not by the methods of inquiry used. (p. 236) Case studies can be very influential in helping to understand the historical background of the topic under study, explore unexpected outcomes, delve into the complexity of interrelationships among people and entities, explore gaps between what is intended and what happens, and in general obtain a comprehensive look at the big picture. Although there are limitations to the case study approach, such as the inability to broadly general – ize, case studies can be particularly useful in gen – erating hypotheses and theories in newer fields (Garson, 2008b). In other words, when we know little about a topic, the case study can be extremely useful in providing context about a new idea and about how variables may affect behavior. Naturalistic Observation The term naturalistic observation typically implies an observational situation where the researcher does not interact in the environment, but merely observes it. However, observation-based research is much more complicated. Naturalistic observa – tion tends to fall into the category of qualitative research, and the goal of qualitative research is to understand human behavior holistically (rather than analytically), and consider the social and cultural context in which we behave (Angro – sino, 2007). The development of systematic observational protocols is a key component in naturalistic observation research, and these studies tend to fall into one of three broad cat – egories: (a) non-reactive (unobtrusive) research, where the researcher does not participate in the events under observation; (b) reactive research, in which the research is immersed and clearly present in the environment but strives for the role of outside observer; and (c) participant research, where the researcher embeds him- or herself into the environment and participants as an active member of the group being studied (Angrosino, 2007). A social worker makes a case study of a neglected child and will record both qualitative and quantitative data. Liquidlibray/Thinkstock lan66845_04_c04_p101-130.indd 119 4/20/12 2:47 PM CHAPTER 4 Section 4.2 Observational Designs Observational research, includ – ing naturalistic observation, strives for some of the same goals as experimental research, but it also faces unique chal – lenges. Validity and reliability are challenges to observational research (Adler & Adler, 1994). Because data recording and interpretation are key in natu – ralistic observation, validity may be threatened by an over – reliance on any one particular approach. One suggestion that Adler and Adler (1994) offered is to employ a multiple observer strategy such that there is not overreliance on any one mem- ber of the research team. Reli – ability is also a concern in nat – uralistic observation, for if a phenomenon is not stable and consistent, then it is difficult to interpret the impact of the singular observation on overall behavior. Observations that are made over varying times and places may yield higher confidence in the reliability of the findings, or as Adler and Adler (1994) put it, “like many qualitative methods, naturalistic observation yields insights that are more likely to be accurate for the group under study and unverified for extension to a larger population. Observations conducted systematically and repeatedly over varying conditions that yield the same findings are more credible than those gath – ered according to personal patterns” (p. 381). So what would a naturalistic observation look like, or, in other words, what are the basic steps that are followed? Angrosino (2007) described a typical sequence of events that is followed in naturalistic observation research: • In the descriptive phase, the researcher is interested in reporting initial observa – tions that are related to the general research questions under study, as well as pro – viding descriptions of the environment being studied, including the people and the place. At this point, observations should be as value-free as possible, without interpretation—the goal would be statements of fact based on direct observation. • Once a broad base is established, the focusing phase begins as researchers strive to sort out relevant observations from irrelevant observations, especially in how these observations relate to the hypotheses and key questions under study. These obser – vations would be more focused on well-defined activities (e.g., traditions, rituals, events) rather than one-time random occurrences. The goal here is to identify pat – terns, especially how the observed patterns relate the research question of interest. • The selective phase might be analogous to a “highway merge” in a large city, where six lanes of traffic are funneled into two lanes. There are still observations to be recorded, but now the key behaviors have been focused on and are under care – ful scrutiny. Although the entire field of the environment is still under observation, selected observations are used to help provide possible explanations for behaviors. A marine biologist observes a shark in the ocean. Naturalistic observation implies that the researcher does not interact with the environment or participants but just watches and records observations. Flirt/SuperStock lan66845_04_c04_p101-130.indd 120 4/20/12 2:47 PM CHAPTER 4 Section 4.3 Archival Research • Finally, by the time the saturation point is reached, no new findings are being dis – covered. The major patterns of behavior are well established and rarely change. Further observation after reaching the saturation point has a relatively low probability of discovering anything new. At this point the data analysis and inter – pretation phase is about to begin. Success during this phase of the observational research will very much depend on the coding scheme used and the researcher ’s attention to detail in following the protocols established before beginning the naturalistic observation. Naturalistic observations can be great sources of new ideas. Take, for example, the study by Chiang (2008) where children with autism were studied using naturalistic observation. Children with autism often can be taught communication skills, but these children often lack the ability to spontaneously utilize verbal and nonverbal skills. The goal of Chiang’s research was to document and categorize the levels of communicative spontaneity in autistic children. Thirty-two diagnosed autistic children, ranging in age from 3 to 16, were videotaped in their natural settings, which included special schools for autistic children, special education classrooms, and general education classrooms. One of the key findings from this research was that autistic students exhibited higher levels of communicative spontaneity in non-symbolic forms (e.g., keeping an item, pushing) than symbolic forms (e.g., writing, speech). These types of results, while interesting on their own merit, can provide fertile grounds for additional researchers to formulate ideas about how to better understand autistic children and their communication patterns. As with every methodological approach to studying human behavior, case studies and naturalistic observations have drawbacks as well. Because the participants studied may not be representative of the population, these types of studies lack external validity— the ability to be generalized beyond the participants studied. Case studies and naturalis – tic observations are typically difficult to replicate unless extreme care has been taken to explicitly record the procedures used. Since these approaches do not follow a true experi – mental protocol (including random assignment), causal inferences are not possible, even those with compelling data. Finally, these types of studies rely on highly skilled research – ers, because the potential for influencing the outcome of the study is great, whether it would be experimenter bias in a case study or experimenter reactivity (Brown, n.d.). 4.3 Archival Research A rchival research is a broad term that can be used to describe a wide range of studies. Essentially, archival research involves analysis of data from existing records that were made in natural settings. For instance, by reviewing records from professional base – ball and basketball championship games, Baumeister (1995) found that home teams are more likely to choke (i.e., perform badly), perhaps due to the burden of high expectation made by playing in front of hometown fans. Riniolo, Koledin, Drakulic, and Payne (2003) used archi – val records from 1912 United States Senate hearings (and from the British Board of Trade) to compare eyewitness accounts of the Titanic sinking to the forensic data we now have, particu – larly examining the claim that the Titanic was breaking apart as it sank. Riniolo et al. (2003), after carefully screening testimony that indicated clear observations, found that 15 of 20 eye – witnesses accurately reported this tragic event. Related to more recent events, Martz, Bod – ner, and Livneh (2009), in using archival data available from the National Vietnam Veterans lan66845_04_c04_p101-130.indd 121 4/20/12 2:47 PM CHAPTER 4 Section 4.3 Archival Research Readjustment Study (a national random sample of over 3,000 veterans drawn from 8.2 million veterans who served in Vietnam), found that for veterans with dis – abilities, teaching them problem- solving skills was beneficial only for veterans with mild to moder – ate disabilities (the intervention was ineffective for those veterans with severe disabilities). These three studies illustrate the versa – tility of archival research. None of these researchers actively col – lected data; these researchers examined baseball box scores, congressional testimony, and pre- existing survey data, respectively. Although there are many different forms of archival research, we’ll end this chapter by reviewing two major approaches to analyzing archival data—content analysis and meta analysis. What is the relative contribution that archival research can make to our under – standing of human behavior? Baumeister (1995) rightly pointed out that archival data may not the best choice for testing and building theories in psychology; the laboratory probably remains the best environment for achieving those goals. However, archival research can be invaluable in providing real-world examples of phenomena that are stud – ied in controlled laboratory settings. As Baumeister put it, “perhaps the best compromise is that these [archival studies] should be regarded as extending, illustrating, and confirm – ing laboratory studies rather than as primary, direct tests of theory” (p. 646). Content Analysis In some ways, content analysis bridges the worlds of qualitative research and quanti – tative research (Duriau, Reger, & Pfarrer, 2007). A typical approach to content analysis would be to identify key textual information. Content analysis takes qualitative state – ments, writings, and other forms of language and quantifies that data. For example, a conceptual analysis approach means that the frequency of representation of a particular word or concept is quantified in content analysis (Busch et al., 2009). Thus, you could examine presidential inaugural speeches and count the number of mentions of freedom, a free society, and so on. After identifying the research question and selecting the samples to be analyzed, the content is coded into predetermined categories, where frequency counts are now possible. A second approach within content analysis is called relational analysis, where other words are found in the text and then examined for their proximity to key concepts being analyzed (Busch et al., 2009). Thus, when freedom is mentioned in inaugu – ral addresses, what other concepts are mentioned within close proximity to mentions of freedom, such as liberty, taxation, or civil rights? Because of the versatility of content analysis, this methodological approach can be used to study a wide variety of topics. For example, Sonpar and Golden-Biddle (2008) looked at Archival research uses data from previous studies. Hemera/Thinkstock lan66845_04_c04_p101-130.indd 122 4/20/12 2:47 PM CHAPTER 4 Section 4.3 Archival Research adolescent theories of cognitive organization and found that content analysis was helpful in elaborating on pre-existing theories. Chancey and Dumais (2009) used content analysis in a more longitudinal approach. Their interest was in how couples who purposely choose to be childless (“voluntarily childless”) have been depicted in marriage and family text – books ranging from the 1950s through 1990s. In this archival research using content analy – sis, Chancey and Dumais discovered that different decades presented different themes. Content analysis revealed that marriage and family textbooks in the 1950s depicted the voluntarily childless as individuals who understood that child-rearing was a challenge and to be feared, and those rejecting becoming parents were preemptively avoiding caus – ing harm in children. In the 1960s, the voluntarily childless were seen as actively pro – moting marital satisfaction, whereas texts of the 1970s tended to say very little about the voluntarily childless. Content analysis of textbooks revealed that in the 1980s there was increased scholarly focus on the voluntarily childless, as well as discovering that there was often a disparity in opinions within a couple of the voluntarily childless decision. In the 1990s, content analysis revealed that the most effort was aimed at dispelling the myths surrounding the voluntarily childless and sometimes characterized these couples’ decisions as brave in the face of societal expectations. Clearly Chancey and Dumais would not be able to go back in time and interview voluntarily childless couples, but the con – tent analysis of marriage and family textbooks over time provided insight into prevalent thought processes at the time. There are a number of advantages available to researchers who elect to use a content analysis approach. This listing combines the advantages as stated by Busch et al. (2009) and Duriau et al. (2007)—content analysis (a) allows for both qualitative and quantitative analysis; (b) can use a longitudinal approach that allows for the examination of historical contexts over time; (c) is an unobtrusive method of analyzing interactions; (d) can provide insight into complex cognitive operations, such as language and thought; and (e) uses an approach that can be replicated by others interested in the same phenomenon. Some of the disadvantages to content analysis include the labor intensiveness of obtaining and coding texts, that it can sometimes operate without a theoretical base, the reduction of complex human thoughts into parsed words, the oversimplification of the complexity of language by its reduction into frequency counts, and even the inability to capture the totality of con – text, such as the historical period or geographical context of what is spoken and written. Even with these drawbacks, content analysis is a powerful tool for the analysis of language. Meta-Analysis Within a single year, multiple studies are published on the same topic, and when thinking about a topic over a long range of time (like you would do if you were reviewing the liter – ature), discrepancies appear in the interpretation of data. For instance, there are claims in the literature that a moderate amount of red wine can be very healthy for you with regard to cardiovascular function, but there are conflicting reports about the effect of red wine, especially if you consider the risk of alcoholism. So if you had to make an overall decision about whether or not to consume red wine, how would you combine the results of the previous studies to make a decision? This is precisely the dilemma than then-senator Wal – ter Mondale faced in 1970 (Walter Mondale later went on to serve as vice president of the United States under President Jimmy Carter from 1977–1981). When Mondale addressed the 1970 convention of the American Psychological Association, he wanted some insight into the impact government programs were having on children’s development, and here’s what he said (Attwood, 2009): lan66845_04_c04_p101-130.indd 123 4/20/12 2:47 PM CHAPTER 4 Section 4.3 Archival Research I had hoped to find research to support or to conclusively oppose my belief that quality-integrated education is the most promising approach. But I have found very little conclusive evidence. For every study, statistical or theoretical, that contains a proposed solution or recommendation, there is always another, equally well documented, challenging the assumptions or conclusions of the first. No one seems to agree with anyone else’s approach. But more distressing, no one seems to know what works. As a result, I must confess, I stand with my colleagues confused and often disheartened. (¶s 18–19) The contribution of meta-analysis is that it pro – vides an approach for combining the results of multiple studies so that general effects can be summarized (Bangert-Drowns & Rudner, 1991). Although the idea of combining the results of stud – ies can be traced back to Pearson and Fisher (Coo – per & Lindsay, 1998), the 1970s saw an enhanced effort to combine studies to answer complex ques – tions. For example, in looking at the results of studies of class size and academic achievement, Glass and Smith (1979) combined the results of 725 studies that estimated that relationship. Smith and Glass (1977) looked at the effectiveness of psychotherapy by examining 833 estimates of the effect, and Hunter, Schmidt, and Hunter (1979) looked at how employment tests may or may not differentiate between white and African-Ameri – can employees. For each of these broad questions (class size, psychotherapy, employment testing), meta-analysis combined the results of hundreds of studies to allow for overall conclusions to be drawn. In fact, Glass (1976) is credited for the term meta-analysis (Cooper & Lindsay, 1998), and Glass (1976) defined meta-analysis as “the statisti – cal analysis of a large collection of analysis results from individual studies for purposes of integrat – ing the findings” (p. 3). There are at least four different types of meta-analyses that a researcher could conduct (Bangert-Drowns & Rudner, 1991; Cooper & Lindsay, 1998), and you might see more than one analytical approach used in a report of meta-analytic findings. Here are brief descriptions of the approaches that are used: • Using the vote count method, the overall impact of the study is counted as a “vote” in showing that the intervention has a positive (beneficial) effect, a nega – tive (detrimental) effect, or no effect at all. The results are tallied to come to an overall conclusion about the topic under study. • The classic method (combining probabilities across studies) might involve the sum – mation of all the p values in used in significance testing, and then using a statisti – cal procedure to determine if the overall result is likely to occur by chance or not. There are numerous studies that are completed and published about the same topic. Using meta-analysis provides an approach that combines the results of these studies into a summary. age fotostock/SuperStock lan66845_04_c04_p101-130.indd 124 4/20/12 2:47 PM CHAPTER 4 Section 4.3 Archival Research Rather than vote count, now each study contributes a degree of impact to the overall conclusion. Since p values are relatively common in published studies, this approach is straightforward, but sometimes overestimates the impact of the effect. • When using the effect size method in meta-analysis, the degree of impact is more precise. With the p value approach above, studies with large Ns are more likely to have small p values—effect size is not as influenced by the N as p values are. There are procedures that can weigh the impact of an effect size when that effect size comes from a larger-scale study. • Meta-analysis can also examine the relative contribution of variables within stud – ies using effect size indices. Sometimes called homogeneity analysis or tests of homogeneity, the researcher can examine the degree to which different aspects of the research design impacted the effect sizes observed. This is a brief overview of the types of approaches available to the researcher utilizing meta-analysis. If you were to perform a meta-analysis, the steps you would follow might look like this: (1) clearly define the hypothesis and the operations under study; (2) conduct a thorough literature search; (3) categorize the studies; (4) transform to a common metric; (5) analyze the data; (6) formulate discussion and draw conclusions; and (7) generate a report (Botella & Gambara, 2006). Even from this description, you can see multiple meta- analytic approaches being used, such as vote counting and calculating effect sizes. One of the benefits of the development and use of meta-analysis is that it has increased the rigor with which research is discussed and published (Hedges, 1990). The meta-analytic approach is not without drawbacks, however (Bangert-Drowns & Rudner, 1991; Nugent, 2009). Three of the most commonly mentioned challenges in meta-analysis are (a) missing data on effect size, (b) missing data on study characteristics, and (c) publication bias. In some cases, published research may not fully present the statistical results needed for a researcher using meta-analysis to be able to calculate an effect size for each study, and in other studies, such as a case study, there may be no statistical information at all. The com – plexity of the design might also make extracting relevant information about the effect size complicated (Hedges, 1990). If too many data are missing from the studies on a particular topic, this poses a threat to the value of the interpretation of the remaining studies. Quality meta-analyses also need information about how the study was conducted, espe – cially information about the procedures used in the study, how participants were assigned to groups, precisely how the dependent variables were measured, and so on (Hedges, 1990). If this type of information is not included in the studies being reviewed, then it is difficult to determine the relative contribution of the study to the meta-analysis. Because journal space is typically at a premium, sometimes key information that meta-analysis researchers need might not be included in published accounts. Another area of concern for meta-analytic research involves what is called publication bias. Typically, to publish an article in the psy – chological literature, you need to reject the null hypothesis—that is, find a significant dif – ference between groups, relationship, or association between variables or predictors of a criterion. Thus, studies that accept the null hypotheses are difficult to publish. A researcher conducting a meta-analysis would prefer to have a complete picture of all the research con – ducted on the topic (regardless of outcome), but this is unlikely due to this publication bias. If what is in the literature is not representative of the totality of the research, then the results of the meta-analysis from an unrepresentative sample must be interpreted with caution. Here is an example of a meta-analytic strategy that demonstrated the benefit of this approach. Blinn-Pike, Worthy, and Jonkman (2007) wanted to arrive at an estimate of the lan66845_04_c04_p101-130.indd 125 4/20/12 2:47 PM CHAPTER 4 Chapter Summary approximate percentage of college students who exhibit a serious gambling disorder. Rather than conduct an entirely new study to examine this issue, they conducted a meta- analysis of 15 college student gambling studies through 2005. One of the beneficial features of studying this topic is that there is a common metric used to measure gambling disor – ders, the South Oaks Gambling Screen (SOGS). Thus, each of the 15 studies had to use (at least) the SOGS as an outcome (dependent) variable. Other screening criteria for inclusion in the meta-analysis included (a) the percentage of disordered gambling in students had to be reported; (b) the study had to come from a peer-reviewed publication or dissertation; and (c) the study had to be con – ducted in the United States or Canada. Blinn-Pike et al. (2007) were able to estimate that the percentage of disordered gam – blers when considering a college student population is 7.89%. The results of 15 studies to arrive at this estimate may pro – vide a more accurate estimate of the overall existence of gam – bling problems among college students as compared to a study conducted on one campus. In the next chapter, we’ll look at a very different type of research design—the single subject design—and we’ll explore the insights achievable from this valuable research method. Chapter Summary R esearch conducted in controlled settings, such as a laboratory, can often lead to powerful conclusions about behavior, and at times may yield insights into cause- and-effect relationships, the gold standard desired by scientists. However, people do not live in laboratories, but rather the real world, so a number of additional research approaches are necessary to better understand complex behaviors influenced by numerous variables in social settings. Quasi-experimental designs offer many different approaches to understanding these potential influences to behaviors, beliefs, and perceptions. Obser – vational designs and archival research strategies can lead us to insightful revelations about the multitudinous factors that influence each of us. A psychologist equipped to conduct research in an applied setting wants to be as prepared as possible, and this means that he or she should have access to multiple methodologies to ensure that the questions of inter – est are answered with the strongest approach available, given the constraints of collecting data in the real world. Blinn-Pike, Worthy, and Jonkman used a meta-analytical strategy approach to analyze 15 studies on college gambling to approximate the percentage of college students with a serious gambling disorder. Hemis.fr/SuperStock lan66845_04_c04_p101-130.indd 126 4/20/12 2:47 PM CHAPTER 4 Chapter Summary Research designs from this chapter Type of StudyDesign Name Symbolic Representation X = treatment or intervention O = observation, dependent variable measurement Brief Features Quasi- Experimental Nonequivalent Control Groups O X O O O This design is superior to the pretest- posttest only design comparing the effect of the independent variable to a control group, but because of the lack of random assignment, the groups may not be roughly equivalent at pretest. Quasi- Experimental Time Series O O O X O O O Repeated observations pre and post the independent variable manipulation allow for the tracking of trends. Quasi- Experimental Cohort N/AThis design tracks a similar group of people over time, such as an age cohort of a generation or a period cohort such as young adults during the time of the Vietnam War. Quasi- Experimental Regression Discontinuity N/AStarting with a large number of individuals, an intervention is applied to some individuals in the group based on a cutoff score, and the key outcome is whether or not the trends occurring before the intervention occur after the intervention (that is, continuous or discontinuous). Observational Case Study N/AThis involves the intensive study of one individual while examining a limited set of specific behaviors. Observational Naturalistic Observation N/AParticipants are studied in their natural environment by the researcher, but the researcher only observes and does not interact with participants. Archival Content Analysis N/AThis set of techniques allows for the analysis of qualitative data such that trends can be analyzed and conclusions can be drawn from the data. Archival Meta-Analysis N/AThis methodology allows for the outcomes of multiple studies on the same topic to be combined meaningfully and statistically to form overall, general conclusions. lan66845_04_c04_p101-130.indd 127 4/20/12 2:47 PM CHAPTER 4 Concept Check Concept Check 1. The following symbols would be described in words as X O O A. two groups; only one gets the treatment, and both get the posttest. B. one group gets two treatments. C. randomly assign participants to two groups; both get treatments, and one gets posttest. D. two groups get the same treatment. 2. In an interrupted time series design, the treatment is done A. before observations. B. in the middle of observations. C. after the observations. D. after a single observation. 3. A group of people who share a critical experience over the same span of time defines a A. time series. B. cohort. C. control group. D. treatment group. 4. According to Greene (2003), the sequence of program evaluation phases is A. causation, implementation, generation. B. causation, generation, implementation. C. implementation, generation, causation. D. generation, implementation, causation. 5. Fundamental lessons for program evaluation (McCartney & Weiss, 2007) include all of the following EXCEPT A. adopt a fair and reasonable scientific explanation. B. use mixed-methods designs. C. discourage critique of the data. D. synthesize all research findings. Answers 1. A. Two groups; only one gets the treatment, and both get the posttest. The answer can be found in the Introduction. 2. B. In the middle of observations. The answer can be found in Section 4.1. 3. B. Cohort. The answer can be found in Section 4.1. 4. D. Generation, implementation, causation. The answer can be found in Section 4.1. 5. C. Discourage critique of the data. The answer can be found in Section 4.1. lan66845_04_c04_p101-130.indd 128 4/20/12 2:47 PM CHAPTER 4 Key Terms to Remember Questions for Critical Thinking 1. Think about the variety of research approaches studied to date. How will you know what research approach is more appropriate than another with a particular research question or population of interest? How do psychologists come to have confidence in making these types of decisions? What will you need to do to build your confidence in your ability to select a methodological approach that matches the research questions you are interested in? 2. As you reflect back on your undergraduate career, you may see now that a particular sequence of courses might have been more helpful than others. Think – ing about quasi-experimental designs, how might you design a study to show that one sequence of prerequisites is superior to a different set of prerequisites? How would you select one quasi-experimental design over another? How would you determine the superiority of one particular course sequence compared to another? 3. You may have conversations from time to time with students who are attending college a different way from how you are attending college currently. You might make the argument that an online education is superior, listing the advantages you have experienced. Another person who attended a “brick-and-mortar” cam – pus may not think as highly of online coursework, and may provide his or her list of advantages for that type of learning experience. How might you design a quasi-experimental study to systematically examine the similarities and differ – ences that exist among online, classroom, and hybrid educational approaches? Key Terms to Remember archival research A research methodology that involves analysis of data from existing records that were made in natural settings. case study A research methodology that focuses on a particular case of interest. This case may be a person, a group, or perhaps an organization. Case studies can utilize qualitative and quantitative methods. causation phase The phase of program evaluation in which the program evaluator attempts to demonstrate and measure the impact of the program. cause and effect An analysis that attempts to examine the causes and results of actions or behaviors. cohort A group of people who have shared experiences over the same span of time. content analysis A method of analysis that takes qualitative statements, writings, and other forms of language and quanti – fies those data. generation phase The phase of program development where a needs assessment may be conducted prior to the implementa – tion of a program, to see if the need justifies the implementation of a program, followed by program development and refinement. implementation phase The phase of program development where the program begins, provides services to clients and the community, and hopefully has a positive impact. meta-analysis A method of analysis that provides an approach for combining the results of multiple studies so that general effects can be summarized. lan66845_04_c04_p101-130.indd 129 4/20/12 2:47 PM CHAPTER 4 Web Resources naturalistic observation An observational situation where the researcher does not interact in the environment, but merely observes it. nonequivalent control groups design A design in which an experimenter is unable to randomly assign participants to the different groups due to external factors. panel studies Studies that are designed to measure the same participants at different points over time. program evaluation An assessment of the impact of a program based on its pre- stated goals. quasi-experiment When a participant is not randomly assigned to a group but instead assigned to a group based on char- acteristics that he or she already possesses. regression discontinuity design A study design in which assignment into the treat – ment group or the control group is based on a predetermined cutoff. time series design A study design that includes consistent and multiple observa – tions of a variable that is repeated a sub – stantial number of times. Web Resources Explanation and examples relating to quasi-experimental design; explains the lack of randomization in these designs and how that can affect research. http://allpsych.com/researchmethods/quasiexperimentaldesign.html Outline and definition of the regression-discontinuity design, which measures the effec – tiveness of a treatment or program that a psychological researcher may be investigating in his or her research project. http://www.socialresearchmethods.net/kb/quasird.php lan66845_04_c04_p101-130.indd 130 4/20/12 2:47 PM 5 Single Subject Designs Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • identify different types of reversal/withdrawal designs and understand the benefits and drawbacks of each. • comprehend the appropriate usage of more complex single-subject designs, such as the mul – tiple baseline and changing criterion designs. • summarize the challenges to analyzing data from single-subject designs as well as appreciate the applications and limitations of these designs. Science Faction/Superstock lan66845_05_c05_p131-156.indd 131 4/20/12 2:48 PM CHAPTER 5 Introduction Introduction A t the close of the last chapter, you read about case studies and archival research. So what’s the difference between a case study involving one participant and a single-subject design involving one participant? In a single-subject design (SSD), there is more experimental control of the presentation and withdrawal of the independent variable levels. That is, the single-subject design can do true experimental manipulations, whereas case studies are typically observation-based or based on archival records. Thus, the advantage of the single-subject design is that cause-and-effect conclusions can be approximated but with limited generalizability since the data are typically based on one participant. The history of the single-subject design comes from the tradi – tions of behaviorism, behavior therapy, and behavior analysis (Freeman, 2003). However, the applications of single-subject designs are diverse and not lim – ited to those with a behavioristic orientation—this methodologi – cal approach has been used in social work, education, cogni – tive rehabilitation, sport psy – chology, counseling, and occu – pational therapy, to name a few examples (Freeman, 2003). SSDs were used to modify the behavior of Little League base – ball coaches (Martin, Thompson, & Regehr, 2004) and to help those with autism develop better independent work and play skills (Hume & Odom, 2007). A machine records brain waves as a participant performs a task on a computer. Single-subject designs allow more experimental control and manipulation of the independent variables than case studies. Age fotostock/SuperStock Voices from the Workplace Your name: Jennifer B. Your age: 35 Your gender: Female Your primary job title: Director of Operations Your current employer: Creative Community Options How long have you been employed in your present position? 1 month What year did you graduate with your bachelor’s degree in psychology? 1994 (continued) lan66845_05_c05_p131-156.indd 132 4/20/12 2:48 PM CHAPTER 5 Introduction Describe your major job duties and responsibilities. Operate a community-based residential program for adults with intellectual and developmental dis- abilities, as well as chronic mental illness. Also operate a supported employment program for the same population groups. Both programs provide support to approximately 140–150 individuals. The operations involve overseeing 10 to 12 coordinators who complete the program planning and provide the direct supervision to the direct support professionals. In that oversight is managing service rates, contracts, ensuring programs are operating within the state and federal regulations, providing support to the coordinators with challenging staff issues as well as concerns with the individuals we provide support to. There is also significant involvement with the budgeting process of the organization, work – ing to develop and maintain relationships with stakeholders of the organization and assisting in team meetings when there are significant challenges in providing service to an individual. What elements of your undergraduate training in psychology do you use in your work? In my current position, one of the programs provides services to individuals with chronic mental ill – ness. My degree is helpful in providing support to the coordinator of those individuals. Many of the intellectually disabled individuals also have varying degrees of mental illness. The oversight, both directly and indirectly of 170+ employees requires regular use of the training I received in my under – graduate work as well. What do you like most about your job? I have been in the human service field since leaving undergraduate school. I have always worked with individuals that have some form of a disability. I have a strong passion to see that individuals with intellectual disabilities are given the same opportunities in their life as those without. I get to support the individuals we serve, through the administration of our programs, in living their life as they want to in their own homes, apartments and in jobs at businesses in their own community. Through my experiences as a direct support professional, program manager/coordinator, and Med – icaid case manager, I can now develop current coordinators and assist them in learning new skills through which will provide the highest of quality of service to the individuals who receive support from our agency. What do you like least about your job? In this field, there are many regulations both federal and through our state which impact the job that we do daily. I feel the regulations are necessary due to past injustices our field allowed to occur. However, many times the regulations go too far and impact the quality of service as we must focus on items which don’t directly impact the people we serve. I now have a strong desire to begin to work with legislators and representatives in an effort to impact the laws and regulations surrounding this field. Beyond your bachelor’s degree, what additional education and/or specialized training have you received? Various training regarding the support of individuals with Autism: TEACCH, PECS, Training in Positive Behavior Support, MANDT, many conferences on supervision of employees What is the compensation package for an entry-level position in your occupation? $28,000–$34,000 salary for a coordinator or case manager. $9.00–$11.00/ hour for direct support staff. What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? Health insurance with dental and vision in many places. 401K. Holidays off—unless you are working as a direct support professional. Vacation, sick time or PTO. Emergency leave time. Flexible hours. Voices from the Workplace (continued) (continued) lan66845_05_c05_p131-156.indd 133 4/20/12 2:48 PM CHAPTER 5 Introduction What are the key skills necessary for you to succeed in your career? Passion for the individuals served; Organization skills; Ability to manage large amounts of paperwork, tight deadlines, and time to handle immediate person-served needs. Soft skills—working well with people, developing resources and information, teaching as opposed to counseling. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? Social Work courses would have been very helpful to me. I had to learn how to develop assessments on the job and didn’t get that in a psychology degree. The assessments in the human service field are looking at the main life domains—where you live, work, your finance situation, social life, spiritual life, etc. Classes to learn how to develop a thorough social history and how to work with parents, guard- ians or family to get that delicate information. For my current place in my career, management classes are key. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? My degree in psychology did not require an internship as the social work degree did. I firmly believe that there should be an internship requirement for a psychology degree. Getting out during under – graduate work and obtaining an entry-level position which doesn’t require a degree is a great way to start to gain experience as well as to identify what field you want to work in. Psychology is a rather general degree at this point, which is a great cornerstone to further graduate work but is starting to be difficult to use in the human service field. If I were to graduate now with my psychology degree, I would have a difficult time getting to this level in my field. Many positions are requiring a social work license. I am fortunate that the state of Iowa doesn’t require it as often as many states do at this time. As an undergraduate, do you wish you had done anything differently? If so, what? I would have kept my study in psychology and sociology; however I would have at least minored, if not had a triple major in Social Work. That would allow me to now obtain my Social Work license. What advice would you give to someone who was thinking about entering the field you are in? Obtain a job with an organization that provides services to individuals with a disability. Get a direct support job and learn the field directly with the individuals we support. They are your greatest teach – ers and will help establish the passion for what we do. Many of the positions in my field require that you have at least a year or two of direct support experience. It is crucial. Attempt to vary the disability ranges. It is best to have experience with individuals that have a brain injury, individuals with an intel- lectual disability and individuals with a chronic mental illness. Substance abuse is also an area of this field which is helpful to have experience with. If you were choosing a career and occupation all over again, what (if anything) would you do differently? Absolutely nothing. I am very fortunate to be where I am at in my career at this time. Every position I have had, has given me critical experience and information which has allowed me to develop to this place in my professional life. Copyright © 2009 by the American Psychological Association. Reproduced with permission. The offi – cial citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychology Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Asso- ciation, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychologi – cal Association. Voices from the Workplace (continued) lan66845_05_c05_p131-156.indd 134 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs 5.1 Reversal/Withdrawal Designs M ost SSDs begin with a phase where baseline data are collected concerning the behavior of interest; this phase is labeled the A phase in a single-subject design. After multiple observations to establish a stable baseline (more on this later), some sort of intervention (i.e., independent variable manipulation) is introduced with the intention of changing the baseline frequency of behavior, either to increase the frequency of a positive behavior or decrease the frequency of a harmful behavior. This introduction of the intervention is called the B phase in an SSD. In some ways, the multiple obser – vations during the A phase (baseline) followed by an intervention B phase resemble an interrupted time series approach (Sharpley, 2003), as mentioned in the previous chapter. The graph in Figure 5.1 presents what typical data would look like from this design—this would be referred to as an AB design (from Sharpley, 2003). The fundamental notion of reversal designs is this application and removal of an interven – tion strategy, or in the case of the AB design, the baseline followed by a dramatic change (a reversal if you will)—the intervention. Even though this methodological approach resem – bles an interrupted time series design, the AB design also shares characteristics of a classic pre-post or before-after design (Taylor & Adams, 1982). The parallels are presented next: Abaseline B intervention O observation Xindependent variable Cigarettes smoked/day Days 12345678 910 50 45 40 35 30 25 20 15 10 5 0 11 12 13 14 15 16 17 18 19 20 A (Baseline) B (Intervention) This is a hypothetical depiction of the observational data from an AB design; the A phase is the baseline, and the B phase is with the intervention applied. Source: Sharpley (2003) Figure 5.1: An AB design lan66845_05_c05_p131-156.indd 135 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs If you think back to our experimental designs, the O X design might tell us something, but not much. A similar conclusion can be made for the AB design. The B phase will let us know about a change in behavior, but it will be unclear about the causal connection of the change—was intervention or some other event in tandem with the intervention respon – sible for the change in behavior? Because of this limitation, most reversal designs contain more than two elements (that is, more than just an A and a B); thus, later in this chapter we’ll examine ABA designs and ABAB designs. The repeated measurement of baseline behavior alternating with interventions gives us greater confidence that the intervention is a cause for behavior change. However, there are some situations were an AB design would be inappropriate. Freeman and Mash (2008) described a situation where, because of ethical considerations, it would be inappropriate to remove an effective intervention, if the B phase shows that the intervention is working, such as in the case of treating a life- threatening illness. A major advantage of single-subject designs is that they allow for the identification of functional (causal) relationships (McReynolds & Thompson, 1986; Taylor & Adams, 1982). Researchers often desire to seek out causal relationships, and it’s the notion of internal validity that addresses the causal relationship between the independent vari – ables and the dependent variable—or in SSD research, the interventions and the base – lines. Taylor and Adams (1982) stated that “single-subject designs demonstrate better control of the factors that can affect internal validity than do group designs: i.e., in single- subject research it is more unlikely that events outside of the experimental manipulation affect the behavior” (p. 96). Another methodological advantage of the SSD is its ability to identify variation in dif – ferent individuals—an idea that is more formally called intersubject variability (Free- man, 2003; Freeman & Mash, 2008; McReynolds & Thompson, 1986). For example, in a classic within groups design, if a significant difference was demonstrated between two groups, this would be indicated by group scores changing over time. However, there might be individuals within a group who did not change. The SSD first examines indi – vidual change; with replications extended to a second, third, and fourth individual, gener – alizations can begin to be made about external validity (McReynolds & Thompson, 1986). Another methodological advantage is the ability to explore intrasubject variability; that is, how a single person can change and vary over time, especially with repeated baselines and interventions (Freeman, 2003). In addition to the methodological advantages of SSDs, there are practical advantages as well. Rapoff and Stark (2008) summarized the advantages: • SSDs offer flexibility; if an intervention is not working, that fact can be quickly identified, and another approach can be used. • SSDs can be widely applied because a large N is not required to conduct the study; individuals with rare problems or disorders can be studied. • When it is unethical to withhold treatment, an SSD can be used to assess the potential benefits of that treatment. • The results may be of greater interest to clinicians, and SSD research is perhaps more “doable” by those working in a clinical setting. • SSDs allow for the gathering of empirical evidence that can help support or refute the effectiveness of specific therapeutic interventions. lan66845_05_c05_p131-156.indd 136 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs Before we can address the more complex SSDs, such as ABA and ABAB, we start with the baseline (A phase) and work to establish a stable baseline. Establishing Stable Baselines Establishing a stable baseline is an important component in the success of a single-subject design study (Freeman & Mash, 2008). The baseline data (A phase) should ideally be stable so that changes in the baseline are obvious once the intervention (B phase) is introduced. Multiple measures of the dependent variable are of interest. A generally accepted mini – mal number of baseline observations is three (Barlow & Hersen, 1984), although there are exceptions in the literature. After multiple measures have been obtained, the next objec – tive is stability. In Figure 5.2, you can see examples of the A baseline phase where the baseline is variable, hence not desired. The problem with a variable baseline is that it will make comparisons to the B intervention phase more difficult. With a stable baseline, any effect of the independent variable inter – vention (B) will be more visible (Freeman & Mash, 2008). See the example in Figure 5.3 for a stable baseline. Once the stable baseline is established, we are ready for more types of reversal designs, including the ABA and ABAB variations. Frequency Sessions 123456 10 9 8 7 6 5 4 3 2 1 0 When using single-subject designs, a stable baseline is needed as a reference point for future comparisons. Here is an example of when a non-stable baseline is present. Source: Freeman and Mash (2008) Figure 5.2: A non-stable baseline example lan66845_05_c05_p131-156.indd 137 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs Frequency Sessions 123456 10 9 8 7 6 5 4 3 2 1 0 This graph depicts a stable baseline, the condition desirable as part of the A phase in single-subject designs. Source: Freeman and Mash (2008) Figure 5.3: A stable baseline ABA: The Withdrawal of Treatment If you understand the basics of what A’s and B’s stand for (A = baseline, B = interven – tion), then you can decipher what it means when an ABA design is presented. There will be three phases in this single-subject design experiment, a baseline phase, followed by an intervention, and then followed by return to baseline. This is an elegant design. First, you establish a baseline of behavior (hopefully stable), needing at least three observations (data points). Then, you introduce the independent variable manipulation—that is, the intervention—which is intended to change behavior as compared to the baseline. So far, this is the AB design, and we can visually inspect the graphical data to see if the B portion is higher (or lower) than the A portion of the data. But then, the intervention is removed. If the intervention is truly the source of changing the behavior from baseline, then remov – ing the intervention may cause a return to baseline levels—thus A phase, B phase, then A phase again. If in the second baseline phase the behaviors of interest do not return to normal, then we know that some other factor is acting along with the intervention to influ – ence the dependent variable scores. Here’s an example of an ABA design study, with real data. Pates, Maynard, and Westbury (2001) worked with three college basketball players to help increase shot accuracy. Over a 4-week period of time, Pates et al. collected accuracy data on each of the players for jump shots and set shots (only the jump shot data are presented below). After 4 weeks of baseline data (A), the authors trained the players using hypnosis. The players were given audiotapes to listen to that taught them a trigger word to use while playing basketball, with the inten – tion that use of the trigger word would recall a hypnotic state of relaxation where the play – ers could concentrate better on the shot being attempted. Accuracy was monitored for the 4 weeks during this “B” phase of the study. Finally, the players returned to baseline (A); the audio training tapes were retrieved, and the researchers verified that the trigger words were lan66845_05_c05_p131-156.indd 138 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs no longer being used by the players. The data from the study are presented in Figure 5.4. Each row of graphs represents the performance of one of the three basketball players. As you can see, the B phase (intervention) brought about marked increases in shooting accuracy, but when the intervention was removed (second A phase), accuracy fell back to baseline levels, especially for the first two players. When the dependent variable measure (in this case, shooting accuracy) returns to baseline after the removal of the intervention, Baseline 1Baseline 2 Treatment Accuracy (%) 70 60 55 50 45 40 35 65 70 60 55 50 45 40 65 30 12 Trials Participant 1 345678 91 0111213141516171819202122232 4 Baseline 1 Baseline 2 Treatment Accuracy (%) 90 80 75 70 65 85 60 12 Trials Participant 2 345678 91 0111213141516171819202122232 4 Baseline 1 Baseline 2 Treatment Accuracy (%) 12 Trials Participant 3 345678 91 0111213141516171819202122232 4 This is an example of an ABA design study working with three college basketball players to help increase shot accuracy. What is significant about ABA design? Source: Pates, Maynard, and Westbury (2001) Figure 5.4: An example of an ABA design lan66845_05_c05_p131-156.indd 139 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs this pattern provides more confidence that the intervention is responsible for changes in behavior, because its delivery and removal is demonstrated by behavior change. Another example of the effective use of the ABA design is by Xu, Gelfer, Sileo, Filler, and Perkins (2008). In this study, the researchers studied children’s social interactions as the variable of interest. After the baseline phase, a classwide peer tutoring intervention was imple – mented and then removed. The results indicated that the classwide peer tutoring program was indeed effective in improving social interactions. When you’re doing a study like these two, and the intervention proves to be effective, you may want to end the study on a B phase rather than an A phase. That is precisely what hap – pens with an ABAB design (which is next). But first, is there ever a BAB design? Yes. There may be occasions where an intervention needs to be the first course of action, such as some – one being admitted to the emergency room, or where it has already been started by another mental health professional (Freeman & Mash, 2008). Once any immediate danger has dissi – pated, it may be possible to remove the treatment and see if the presenting problem reoccurs. If it does, treatment can be reapplied (i.e., BAB). In fact, Freeman and Mash (2008) suggested that the BAB design may be superior because it ends on a treatment phase. But there are other SSDs that end on the treatment/intervention phase (B), such as the ABAB design. ABAB: Repeating Treatments The ABAB design is a versatile design and is widely applied in numerous areas where behavior change is of central interest. Figure 5.5 presents a graphic of what a typical ABAB design outcome looks like (from Taylor & Adams, 1982). Percent of Time Spent on-Task Sessions Baseline Intervention Intervention Baseline 100 500 This is a hypothetical example of what an ABAB or repeating treatments design would look like. Source: Taylor and Adams (1982) Figure 5.5: An example of ABAB repeating treatments design lan66845_05_c05_p131-156.indd 140 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs The first A is the baseline, followed by the B intervention, followed by withdrawal of the intervention (returning to baseline—A), and then followed by the re-administration of the intervention again (B). This repeated administration and withdrawal of the intervention (independent variable) allows for greater confidence that the B phase is causing a change from baseline. There are good examples of where an ABAB design was successfully employed. For example, Hetzroni and Shrieber (2004) worked with three junior high students with writ – ing disabilities, and they used an ABAB design to look at how word processing on a computer might impact spelling errors, reading errors, and the number of words written per text (the dependent variables). Hetzroni and Shrieber found that by using (and with – drawing) word processing capabilities, during the B phases spelling mistakes and read – ing errors decreased, but the number of words written per text remained unchanged. In a different study using an ABAB design, Finley and Cowley (2005) worked with a 44-year old female to help reduce the amount of time between attempting to go to sleep and falling asleep (that time interval is called sleep onset). By using an intervention (B) that involved going to bed at a regular time each night, and by employing an ABAB design, Finley and Cowley found that the latency of sleep onset decreased substantially. In a third example, Woodard, Groden, Goodwin, and Bodfish (2007) used an ABAB design with autistic children in hopes of treating problem behaviors and core symptoms of autism. The B intervention was dextromethorphan, which is an active ingredient in many cough medications sold over the counter. Testing eight participants, Woodard et al. found that the use of dextromethorphan in an ABAB design was effective for three of them. This type of finding is important because it demonstrates the individual differ – ences that are detectable with an SSD. Had this been a group study, combining the data of all the autistic children may have resulted in failing to reject the null hypothesis. But by using the ABAB design, a more complicated picture emerges: This treatment modality works for some, but not for others. The ABAB design, as versatile as it is, may not always yield a practical application. For example, when employing an ABAB design in sport psychology, if the first administra – tion of the treatment (B) is effective, a participant in the sports world may be reluctant to remove the treatment and return to baseline (A) (Bryan, 1987). Another concern is when the first treatment (B) involves training on some sort of skill. Say, for example, a discus thrower records baseline throws (A) and then is shown a new spin technique to be used right before release (B). With the success followed by the first B intervention, it may be impossible to “unlearn” or remove the training in an attempt to return to baseline. With the ABAB design ending on an intervention (B) phase, this hopefully facilitates the con – tinuation of any positive impact of the intervention. These reversal designs demonstrate the basic versatility of single-subject designs, but as you can imagine, there are more complicated variations on this theme. Two of the themes we’ll explore next are the multiple baseline designs and the changing criterion (interac – tion) designs. lan66845_05_c05_p131-156.indd 141 4/20/12 2:48 PM CHAPTER 5 Section 5.1 Reversal/Withdrawal Designs Case Study: Hey, I’m an N = 1: Changing My Own Behavior Sometimes in learning about theoretical concepts or behavioral principles we might forget that these principles not only apply to our understanding of others, but they also apply to our understanding of ourselves as well. The behavior change principles described in this chapter that can be used for indi – viduals ( N = 1) or small-group (small N) research are principles that we could elect to use to change our own behavior, if we wanted to. By teaching SSD principles in a accelerated course format (6-week class meeting 4 times a week), David Morgan (2009) had his students learn about SSDs by doing an experi – ment—on themselves. Students in the course were able to select a behav – ior that they would like to change about themselves, a behavior that was relatively easy to observe and record. You can probably imagine some of the behav – iors selected by students—amount of exercise, caf – feine consumption, dietary habits, and so forth. In the example highlighted here (with real data), a student’s targeted behavior was to reduce the amount of snack – ing after dinner, more formally known as post-meal consumption. To establish the baseline (A) phase, the student recorded the number of calories per day consumed after dinner for 14 days—the baseline condition presented in Figure 5.6. The solid horizon – tal line across the chart indicates the mean (average) level of calories, just about 500 calories of post- meal consumption per night. The dotted horizontal lines represent a 2 standard deviation distance away from the mean—both the higher distance and the lower distance. Using an SSD approach, if any subsequent observations (after the baseline) fall outside of the dotted range, that is interpreted as a statistically significant behavior change. As you can see from the data, this student reached that signifi – cant level of change on Day 20, and the change was in the desired direction (fewer calories post-meal), and the trend out to Day 29 provides some evidence that the intervention (B) phase was successful. Evening Calories Consumed Successive Days Baseline Intervention Student Calorie Consumption 123456789 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 800 700 600 500 400 300 200 100 0 This graph shows the data from David Morgan’s experiment. What might be significant about the results of this experiment? Source: Morgan (2009) Figure 5.6: An applied example of statistical process control Flirt/SuperStock (continued) lan66845_05_c05_p131-156.indd 142 4/20/12 2:48 PM CHAPTER 5 Section 5.2 The Multiple Baseline Approach 5.2 The Multiple Baseline Approach O ne of the general concerns about SSDs (and about all experimental designs) is the ability to demonstrate internal validity—that is, that the manipulations of the inde – pendent variable caused changes to the dependent variable. In a single-subject design, there are so many factors not under the control of the researcher that it is difficult to have high confidence in a direct causal link, even if the graphical data support that conclu – sion. One methodology available in SSDs is the multiple baseline design . There are at least three different types of multiple baseline designs: (a) multiple baselines across participants, (b) multiple baselines across variables, and (c) multiple baselines across situations (Dattilo & Nelson, 1986; Gliner, Morgan, & Harmon, 2000; Horn & Heerboth, 1982; Robison, Mor – ran, & Hulse-Killacky, 1989). A common example is the multiple baselines across partici – pants. In the hypothetical example from Dattilo and Nelson (1986) in Figure 5.7, all three participants begin in a baseline condition (the A phrase). The intervention (the B phase) is only introduced to one of the participants, while the remaining two stay in the A phase. If These techniques are used in more than just SSD research. This type of approach, including the dotted horizontal lines indicating a distance of 2 standard deviations, is also used in the manufacturing world for the analysis and interpretation of data—it’s called statistical process control (SPC; Morgan, 2009). SPC procedures use charting techniques such as those used here in manufacturing as a means of measuring quality control, such as the quality of product coming off an assembly line. This approach provides a data analytic strategy that is not so dependent on the subjective judgment of interpreting a graph, yet the inclusion of a graph tells a powerful story about whether the behavior change approach works or not. Critical Thinking Questions 1. The research projects that Morgan (2009) described are part of an intense course in Research Methods—and the student project itself took 29 days to complete. If you were enrolled in a 5-week course, for example, do you think you could do this type of project? What modifications would you need to make for it to be successful? For instance, a 5-week course spans 35 days— could you make a significant behavior change in a 35-day span of time? Would that behavior change be long-lasting? How would you measure the long-lasting impact of the behavior change? 2. Small-N designs are intended to help change the behavior of one person or a small number of individuals. If you were to embark on a behavior change project intended to change one of your behaviors, what behavior would you select? How would you make that behavior easy to observe and easy to record? Can you think of practical applications of SSD principles in your daily life? Do you have a son or daughter, another loved one, a neighbor, or a coworker who you would like to see change a particular behavior? What behaviors would you target, and how would you make those behaviors observable and recordable—that is, measurable? 3. Think about the applications of statistical process control and quality assurance. If you have heard about TQM (total quality management) before, then you may be familiar with SPC. How might you use these SSD principles to improve product manufacturing? What would be measured, and what would those graphs look like? In the manufacture of automobile axles, for example, if SPC were used to record the number of faulty axles produced per day (that is, errors), two standard deviations away from the mean might be an acceptable error rate. But what if your company were manufacturing artificial heart valves used in heart bypass surgery or the artificial bone and socket used in hip replacement surgery? How might we change the level of acceptable errors in regard to a single subject design/SPC approach? Case Study: Hey, I’m an N = 1: Changing My Own Behavior (continued) lan66845_05_c05_p131-156.indd 143 4/20/12 2:48 PM CHAPTER 5 Section 5.2 The Multiple Baseline Approach the intervention is effective, in theory only the first participant should be affected, and the other participants see a change in behavior only after the later, staggered introduction of the intervention/B phase. Although this example uses different individuals to receive the interventions, there could be different variables studied in each of the three conditions with one individual (Robi – son et al., 1989). For example, if the multiple baseline design is being used to help a child focus on math skills, reading speed, and positive playground behavior, then three base – lines would be established. Then the intervention would be applied to the math skills first (while not trying to affect reading speed or positive playground behavior). If each behavior improves with the onset of the intervention, this outcome lends greater support to the tentative conclusion that the intervention is causing the change in behavior. Simi – larly, there could be three different scenarios or environments where behavior change is desired, such as home, school, and playground. Using the multiple baseline approach, the intervention would be introduced one environment at a time to determine if the interven – tion alone is responsible for any observed behavior change. Behavior Score Behavior Score Behavior Score Observation Period Baseline Subject A Subject B Subject C Intervention 51 01520 30 20 10 0 51 01520 30 20 10 0 51 01520 30 20 10 0 A hypothetical example of a multiple baseline approach from Dattilo and Nelson. Source: Dattilo and Nelson (1986) Figure 5.7: An example of a multiple baseline approach lan66845_05_c05_p131-156.indd 144 4/20/12 2:48 PM CHAPTER 5 Section 5.3 The Changing Criterion Design The multiple baseline design is essentially the AB design mentioned earlier in reversal designs, but with variations on the theme. One advantage of the true AB design is that if an intervention (B) is found to be effective, it is not withdrawn (Gliner et al., 2000). How – ever, the general drawback to the AB design is that it may be unclear if the B intervention alone is responsible for behavior change; the multiple baselines do strengthen that argu – ment, however. Horn and Heerboth (1982) suggested that, if possible, the combination of the multiple baseline design and the ABA design would lead to very strong conclusions, but if the intervention (B phase) is effective, depending on the situation, it may not be ethi – cally prudent to withdraw an intervention or treatment that works. 5.3 The Changing Criterion Design T he multiple baselines design is essentially an AB design, with the twist being that there are multiple baselines for comparison (different participants, settings, or behaviors). The changing criterion design is also an AB design, except in this case there are not multiple baselines, but the B phase is manipulated such that criteria for opti – mal performance are gradually changed over time (Foster, Watson, Meeks, & Young, 2002; Freeman, 2003; Rapoff & Stark, 2008), which is reminiscent of shaping in operant condi – tioning. So, in the example shown in Figure 5.8 from Foster et al. (2002), baseline data are collected in the A phase as usual. The goal of this particular changing criterion design was to improve the amount of classroom homework that a student was completing. With the baseline around 20-30%, it would be a daunting task for the student to leap from 30% completion to 100% completion, so the intervention is introduced gradually, over time, but adjusting the criterion for success. Percentage of Work Completed Days Changing-Criterion Design BaselineC riterion 1 Criterion 2 Criterion 3 100 90 80 70 60 40 30 20 10 50 0 123456789 10 11 12 13 14 15 16 In this example the goal is to improve the amount of classroom homework that a student is completing. What impact do you think the gradual introduction of the intervention might have had on the data? Source: Foster et al. (2002) Figure 5.8: A changing criterion design example lan66845_05_c05_p131-156.indd 145 4/20/12 2:48 PM CHAPTER 5 Section 5.3 The Changing Criterion Design So as you can see, in the first B (intervention phase), the goal is to get the criterion of class – room homework completed up to 50%; once there and stable, the criterion is increased again. This gradual increase in the criterion over time helps to make the desired goal more attainable. The changing criterion design can also be used to decrease the frequency of an undesir- able behavior, such as cigarette smoking. In the graphic shown in Figure 5.9, Taylor and Adams (1982) utilized a changing criterion design to gradually decrease the number of cigarettes smoked. As you may be aware, it is difficult for smokers to go “cold turkey” and completely quit and maintain that behavior. The changing criterion design is similar to operant conditioning and shaping, where the goal behavior is gradually acquired over time by changing the targeted amount of the criterion variable. In the changing criterion design, the internal validity (causality) argument is strengthened by the occurrence of two events—a change from baseline to intervention and a continuing change in the intervention (B phase) as behavior changes (Freeman, 2003). Number of Cigarettes Smoked Sessions Baseline Intervention 40 30 20 10 0 In this example, a changing criterion design is used to gradually decrease the number of cigarettes smoked. How is the changing criterion design similar to operant conditioning and shaping? Source: Taylor and Adams (1982) Figure 5.9: Another example of a changing criterion design lan66845_05_c05_p131-156.indd 146 4/20/12 2:48 PM CHAPTER 5 Section 5.4 Data Analysis and Evaluating Change 5.4 Data Analysis and Evaluating Change A s in any other research approach, the definition and measurement of the depen – dent variables in SSDs is key. Horner et al. (2005) summarized the features needed of dependent variables in single-subject designs: • The dependent variables are operationally defined such that valid and reliable measurements are possible as well as replication. • The dependent variables are repeatedly measured within the relevant phases (A baseline phase, B intervention phase) such that stable patterns emerge and are recorded. • The definition of the dependent variables remains constant over the course of the study, so that there is no measurement drift in the recording of scores. • The researcher selects variables based on their social significance—that is, the desired change in behavior leads to an improvement of the human condition for the person under study. The measurement of the dependent variable is then followed by an appropriate level of analysis. A long tradition in the analysis of SSD data is by visual inspection (Ottenbacher, 1990; Tankersly, McGoey, Dalton, Rumrill, & Balan, 2006), such as the graphical displays presented at various points in this chapter. Statistical analyses have emerged that go beyond the visual inspection of the data, and these analyses tend to fall into one of two cat – egories: non-regression methods and regression methods (Campbell, 2004; Jenson, Clark, Kircher, & Kristjansson, 2007; Olive & Smith, 2005). An in-depth presentation of each of these approaches is not possible here. A typical approach of the regression method is to calculate a regression line for the baseline data, calculate another regression line for the intervention data, and then make a specific comparison between the two lines (Olive & Smith, 2005). Examples of effect size calculations that do not rely on the regression approach include the standard mean difference, per – centage of non-overlapping data, mean baseline reduction, and per – centage of zero data (Campbell, 2004; Olive & Smith, 2005). Before we leave the topic of ana – lyzing single-subject design data, two additional points are relevant. In addition to statistical vigor in determining the impact of the results from an SSD, “replicability is the final arbiter of whether an effect is likely to occur by chance” (Crosbie, 1999, p. 105). A distinct Statistical vigor and replicability are important factors in single-subject designs. Why do you think these are particularly important to SSDs? PR Newswire/Associated Press lan66845_05_c05_p131-156.indd 147 4/20/12 2:48 PM CHAPTER 5 Section 5.5 Applications and Limitations of Single-Subject Designs advantage of the ABAB design, for example, is the built-in replication of the effect of the treatment and the return to baseline. If confidence in the effectiveness of the interven – tion is desired, repeat the intervention—either within the design or with another similar participant. The validity of the results is established over a series of studies. Finally, the applied researcher using an SSD has to consider clinical significance in addition to statistical significance (Tankersley et al., 2006). That is, helping a person who is trying to reduce the frequency of cigarette smoking, and analyzing those efforts via some of the options described here, may not yield a statistically significant finding (reducing the number of cigarettes smoked per day from 15 to 11), but if the person is helped to some degree, that outcome may have clinical significance to the therapist and the client. 5.5 Applications and Limitations of Single-Subject Designs I n some ways, it’s all about having the right tool for the right job. Single-subject design methodology is extremely effective when used properly in the appropriate situation. In fact, SSDs have been used in many different situations, including school settings, recycling and anti-littering programs in communities, occupational safety, workplace quality and productivity, a variety of clinical disorders, and the treatment of individuals with a developmental disability (Carr & Austin, 1997). There is elegance to the logic of the single-subject design; although sharing some components of a traditional experimental design, the SSD achieves validity and reliability through alternative mechanisms. Many of the examples used throughout this chapter have been of hypothesized data; as you can imagine, actual data from an SSD can be much more complicated. Just to give you a realistic taste, Figure 5.10 presents data from Hume and Odom (2007) where the desired outcome was to enhance work and play systems of three students with autism. To quote from the abstract, “observational data indicated that all students showed increases in on-task behavior, increases in the number of tasks completed or play materials utilized, and reduction of teacher prompts. The results were maintained through the 1-month follow-up” (Hume & Odom, 2007, p. 1166). Not only were the results verified through an ABAB design, but they were also replicated across 3 students, and the benefits of training were still evident after a 1-month follow-up. It is clear that the SSD can be an invaluable tool for detecting change in behavior over time, especially when the focus is on the individual. lan66845_05_c05_p131-156.indd 148 4/20/12 2:48 PM CHAPTER 5 Section 5.5 Applications and Limitations of Single-Subject Designs Percent of Intervals Mark Session Baseline 1 Intervention 1 Training criteria met On Task Prompted Intervention 2 Maintenance Baseline 2 100 90 80 70 60 40 30 20 10 50 0 123456 78910111213141 516 17 18 19 20 21 22 23 24 1 Month Percent of Intervals Scott Session Baseline 1 Intervention 1 Training criteria met On Task Prompted Intervention 2 Maintenance Baseline 2 100 90 80 70 60 40 30 20 10 50 0 123456 789101112131 41516 17 18 19 20 21 22 23 24 1 Month Percent of Intervals Chris Session Baseline 1 Intervention 1 Training criteria met On Task Prompted Intervention 2 Maintenance Baseline 2 100 90 80 70 60 40 30 20 10 50 0 12345 67891011121 314151 6 17 18 19 20 21 1 Month More examples of real data from SSD studies. Note how it is obvious from visual observation alone that a procedure is effective. This feature makes SSD approaches attractive to a large number of researchers. Source: Hume and Odom (2004) 5.10: Observational value of single-subject design data Newman and Wong (2004) summarized some of the chief limitations of SSD methodology. First, in order for researchers to be confident of behavior change, stable baselines must be achieved, and in a clinical setting it may not be prudent to wait for this to happen. Second, as part of a reversal design, a treatment is withdrawn to ascertain its effective – ness. Although this is important for the SSD researcher, this may not be a desired turn of lan66845_05_c05_p131-156.indd 149 4/20/12 2:48 PM CHAPTER 5 Section 5.5 Applications and Limitations of Single-Subject Designs events for the patient or client who is being helped by the intervention. A third concern rests with the analysis of the data (Newman & Wong, 2004). Although more sophisticated types of analyses are now available (regression- and non-regression-based approaches), the clinician may not be overly motivated (or trained) to pursue these higher-level analy – ses beyond visual inspection of the graphical data. Ultimately, the more these statistical approaches are emphasized during the graduate school training of clinicians and practi – tioners, the more frequently these data analytic techniques will be employed. When clear, the graphical picture tells a compelling story; when the picture is foggy, more sophisti – cated data analysis and interpretation techniques are highly valued. Classic Studies in Psychology: Helping Hyperactive Children (Ayllon, Layman, & Kandel, 1975) Since the mid-1970s, amphetamines such as Ritalin have been used to help control hyperactivity in children. Hyperactivity is typically defined as excessive movement, behaving unpredictably, a lack of awareness of the consequences of behavior, the inability to focus, and poor academic performance (Ayllon, Layman, & Kandel, 1975). In the United States in the mid-1970s, about 200,000 children were taking prescription amphetamines for hyperactivity. Using a single- subject multiple baseline design, these researchers were interested in measuring the hyperactivity levels on and off the prescription drug, and the impact that would have on math and reading per – formance. In addition, Ayllon et al. (1975) utilized a reinforcement program during one phase of the design to see if reading and math improvements could be evidenced in hyperactive children while not taking a prescription medication such as Ritalin. The SSD comprised four phases: Phase 1: On medication—three students (Crystal, Paul, and Dudley) who were hyperactive and on medication were studied to establish baseline levels of both hyperactivity and academic performance. Even with medication, hyperactivity is not eliminated; each student exhibited measured levels of still behaving in a hyperactive manner about 20% of the time. This phase established the baseline. Phase 2: With parental and school consent, the prescription medications were discontinued for 3 days to allow the drugs to “wash out” of the students’ systems. As you can see by the graphs below, hyper – activity skyrocketed, with academic performance remaining at its previously low level. Phase 3: Although the children continued without their medication, a reinforcement program (similar to a token economy) was introduced for math instruction only. Phase 4: The three children were still off their medication, but in this phase reading instruction was added to the reinforcement program already in use with math instruction. Study Figure 5.11 and see what happened to student performance over the course of the study. The solid lines connecting filled dots represent hyperactivity. The patterns are roughly the same for all three children. The medication was effective in controlling the hyperactivity, but academic perfor – mance remained low. When the medication was removed, we see the spikes in the solid lines—hyper – activity increased when the drug used to control hyperactivity was removed. But then look what happened when the reinforcement program for math started in Phase 3—hyperactivity dropped, but academic performance improved, and this improvement was sustained in Phase 4 when the reinforce – ment program was extended to reading instruction. Associated Press (continued) lan66845_05_c05_p131-156.indd 150 4/20/12 2:48 PM CHAPTER 5 Section 5.5 Applications and Limitations of Single-Subject Designs So what was so effective about this reinforcement program? The teachers used a token economy. In the classroom, when a child performed a desired behavior, the teacher placed a checkmark on an index card for that child. At the end of the day, students could trade in their index cards with the checkmarks (ranging from 1 checkmark to 75 checkmarks) for items determined to be positive rein- forcers, such as candy, school supplies, free time, eating lunch in the classroom (rather than the caf- eteria), and picnics in the park. The results of the Ayllon et al. study (1975) have lasting implications (Brown & Borden, 1986; Moeller, Barratt, Dougherty, Schmitz, & Swann, 2001; Pelham, Wheeler, & Chronis, 1998) and are meaningful today. One of the conclusions drawn by Ayllon et al. (1975) was that “the present results suggest that the continued use of Ritalin and possibly other drugs to control hyperactivity may result in compliant but academically incompetent students” (p. 144). It is the ingenuity and cleverness of the researchers using a single-subject design that allows for such meaningful conclusions. In fact, Ayllon et al. (1975) concluded their article with this advice: On the basis of these findings, it would seem appropriate to recommend that hyperactive children under medication periodically be given the opportunity to be drug-free, to mini – mize drug dependence and to facilitate change through alternative behavioral techniques. While this study focused on behavioral alternatives to Ritalin for the control of hyperactiv – ity, it is possible that another drug or a combination of medication and a behavioral pro- gram may also be helpful (p. 145). Critical Thinking Questions 1. Thinking about the nature of individual differences observed in this classic study, and then think – ing about a typical grade-school teacher that might have 28 pupils in the classroom, Math (%) 100 50 25 75 0 Reading (%) 10050 25 75 0 12 Medication Days (in Blocks of 3) No Medication Reinforcement MathPlus Reading Math 3456 Crystal Hyperactivity Academic Performance 789 10 11 Math (%) 100 50 25 75 0 Reading (%) 10050 25 75 0 12 Medication Days (in Blocks of 3) No Medication Reinforcement MathPlus Reading Math 3456 Paul Hyperactivity Academic Performance 789 10 11 Math (%) 100 50 25 75 0 Reading (%) 10050 25 75 0 12 Medication Days (in Blocks of 3) No Medication Reinforcement MathPlus Reading Math 3 456 Dudley Hyperactivity Academic Performance 789 10 11 Here are real data from Ayllon et al. (1975). What is the significance of these results? Do you agree with the advice given by Ayllon et al. after their study? Source: Ayllon, Layman, and Kandel (1975) Figure 5.11: The classic SSD design from Ayllon, Layman, and Kandel (1975) Classic Studies in Psychology: Helping Hyperactive Children (Ayllon, Layman, & Kandel, 1975) (continued) (continued) lan66845_05_c05_p131-156.indd 151 4/20/12 2:48 PM CHAPTER 5 Chapter Summary The single subject design is effective at helping to understand the behavior of individuals, which has high utility in psychology. But when the attitudes and opinions of large num – bers of individuals is of interest, surveys and questionnaires can also be effective tools for answering particular research questions of interest; this is the topic of the following chapter. Chapter Summary T he single-subject design approach is so different from, yet so similar to, previ – ous research approaches presented in this book. It is different from much of the traditional research in psychology because it utilizes an idiographic approach to studying single individuals or small numbers of individuals (sometimes called small- N studies). It is similar in that some of the same experimental design principles, such as pretest-posttest designs and time series designs are incorporated into SSD research (but the processes are labeled differently). The ability to make an experimental effect appear and reappear with predictability lends support to cause-and-effect conclusions, which may be extremely important to an individual undergoing treatment but of little generalizability beyond the person studied. However, multiple SSD studies over time can contribute to our cumulative knowledge of effective behavior change. Single-subject designs are quite versatile and are helpful to the practitioner seeing a client or patient, with the goal of affecting change (and being able to graphically represent that effective – ness). Often, with special needs children, the SSD approach is the only type of research approach available for effective use. how might an understanding of SSDs apply to teaching? Does a one-size-fits-all strategy appear likely to work based on an examination of individual differences? Which concepts emerging here might be helpful to curb undesired behaviors in the grade-school classroom? 2. Imagine you were in the midst of this classic study, and following the A (baseline) phase and the B (intervention) phase you discovered a wildly successful approach for helping children with ADD. What would be the ethical considerations of stopping the study there (AB), or continuing on with an ABA or ABAB design? Why would you want to remove an intervention (the B phase) after you had evidence that the targeted behavior change was working? 3. Think about the dynamics of research in a grade-school classroom. What ethical issues might emerge if a teacher who is informed about SSDs attempts behavior change with kids? Is all teach- ing about behavior change, or should some attempts at behavior change be monitored and approved by school officials and parents? What types of bad habits might children pick up in the course of being trained toward good habits (for example, paying junior high school students $10 for every A on a semester report card)? Classic Studies in Psychology: Helping Hyperactive Children (Ayllon, Layman, & Kandel, 1975) (continued) lan66845_05_c05_p131-156.indd 152 4/20/12 2:48 PM CHAPTER 5 Concept Check Research designs from this chapter Type of StudyDesign Name Symbolic Representation A = observation/ data collection B = intervention Brief Features Single-Subject Withdrawal of Treatments A B A With this design, cause-and-effect conclusions are possible when a stable baseline is established. This design is functionally equivalent to a pretest-posttest design, but only with one individual (or just a few individuals, studied separately). Single-Subject Repeating Treatments A B A B The repeating treatments design is similar to ABA, but the intervention is then applied for a second time. This allows for confirmation (replication) that the treatment/ intervention (B) is related to changes in the outcome (A). Single-Subject Multiple Baseline N/AThis design involves the AB pairing of events in multiple individuals, but the start of the intervention (B) phase is staggered so that the direct A-B behavior change is visible at different times across multiple individuals. Single-Subject Changing Criterion N/AThis design builds on the basic AB sequence of single-subject designs, but the level of B outcomes is expected to change over time. That is, there is a sliding scale of expectations about the intended increases or decreases in targeted behaviors. Concept Check 1. A case study differs from a single-subject design in that the single-subject design A. uses only one participant or limited unit. B. randomly selects the participant(s). C. includes the administration of a treatment. D. can be used with observations or interviews. lan66845_05_c05_p131-156.indd 153 4/20/12 2:48 PM CHAPTER 5 Questions for Critical Thinking 2. Ms. Turner has her tennis team practice for a week while she records their per – formance. She then has them practice with feedback from a professional tennis player each day and records their performance for a week. Ms. Turner then has the team return to regular practice while recorded for another week. This single- subject design would be symbolized as A. BAB. B. OXO. C. XOX. D. ABA. 3. The horizontal dotted lines in graphs usually represent A. the introduction of the treatment. B. the averages of the data points. C. 2 standard deviations from the mean. D. where the behavior has changed. 4. The dependent variable in a single-subject design should A. measure between select treatment phases. B. be measured until a “drift” pattern emerges. C. select variables based on their research simplicity. D. use variables that improve the human condition. 5. In the United States in the mid-1970s, about how many children were on pre – scription amphetamines for hyperactivity? A. 20,000 B. 2000 C. 200 D. 200,000 Answers 1. C. Includes the administration of a treatment. The answer can be found in the Introduction. 2. B. ABA. The answer can be found in Section 5.1. 3. C. Two standard deviations from the mean. The answer can be found in Section 5.1. 4. D. Use variables that improve the human condition. The answer can be found in Section 5.4. 5. D. 200,000. The answer can be found in Section 5.5. Questions for Critical Thinking 1. How successful would it be to implement a behavior change program following SSD principles on yourself? Ever kept a tally of number of text messages per day, number of diet soft drinks consumed, number of stairs climbed per day, etc.? How might you use the baseline data on yourself to implement an SSD behavior change program on yourself? What approach would be most effective, in your opinion, and why? If you were in the midst of an ABAB design, and after the first intervention (B), you found that you were seeing substantial improvement in lan66845_05_c05_p131-156.indd 154 4/20/12 2:48 PM CHAPTER 5 Key Terms to Remember behavior areas you selected, would you be willing to return to baseline to exam – ine the potential cause-and-effect relationship? What about the placebo effect? 2. Think about how you might implement an SSD program for your child, a co- worker, or perhaps a neighbor. What behavior would you strive to change? How would you obtain baseline observations on that behavior? How long should the baseline proceed? What types of ethical implications may be in play when observing others prior to the implementation of an SSD research project? 3. Why is it so important to measure change? As a psychology major, will your goal always be to change behavior? What others goals might you pursue? To measure a change in behavior, you have to be able to measure. What types of behaviors would you be interested in changing, either your own or others’ around you? How would you measure that change? Thinking beyond asking questions on a survey (which is the topic of the next chapter), how can we measure behaviors (as opposed to attitudes, opinions, or perceptions)? Key Terms to Remember A phase A single-subject design where baseline data are collected concerning the behavior of interest. ABA design A single-subject design experiment with a baseline phase, fol – lowed by an intervention, and then fol – lowed by return to baseline. ABAB design A single-subject design experiment where the first A is the base – line, followed by the B intervention, followed by withdrawal of the interven – tion (returning to baseline—A), and then followed by the re-administration of the intervention again (B). This repeated administration and withdrawal of the intervention (independent variable) allows for greater confidence that the B phase is causing a change from baseline. B phase The introduction of an interven – tion in a single-subject design. baseline Data that are collected at the beginning of an experiment to determine a starting point in the data collection process. changing criterion design A study design in which different participants, settings, or behaviors change gradually over time. intersubject variability The ability of a single-subject design to identify variation in different individuals. multiple baseline design The study design in which there are multiple base – lines for comparison. reversal designs A study design involving the application and removal of an inter – vention strategy. single-subject design A study design in which cause-and-effect conclusions can be approximated with limited generalizability since the data are typically based on one participant, who serves as his or her own control. stable baseline When a participant’s behavior is consistent previous to par – ticipation in a study to ensure accurate measurement once an intervention is administered. lan66845_05_c05_p131-156.indd 155 4/20/12 2:48 PM CHAPTER 5 Web Resources Web Resources Discussion of the five characteristics of single-subject design, providing examples with visual representations of design methodology. http://www.practicalpress.net/updatenov05/SingleSubject.html Explanation of details of ABAB single-subject design with visual and situational exam – ples. This enables researchers to conceptualize the differences between this research design and other more commonly used formats. http://allpsych.com/researchmethods/ababdesign.html Single-subject design examples with a focus on ABAB withdrawal design, multiple-base – line design, and alternating treatment design. http://winginstitute.org/Graphs/Mindmap/Single-Subject-Design-Examples/ National Center for Technology Innovation, where students can learn the elements of single-subject research, the importance of single-subject research, and real-life application. http://www.nationaltechcenter.org/index.php/products/at-research-matters/ single-subject-research/ lan66845_05_c05_p131-156.indd 156 4/20/12 2:48 PM 6 Survey and Questionnaire Research Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • understand the decisions that are made regarding how the population is sampled and the various techniques to approximate a representative sample. • compare and contrast different survey research methods and comprehend what research situ – ations match better with different research methodologies. • appreciate different survey research designs and the various scaling methods that can be used to construct survey items. • anticipate the types of errors that may occur within the survey research project, know how to handle data collection issues, and begin to understand the various approaches to analyzing the data collected. • construct survey items using the appropriate scale that helps to capture the desired behav – iors, perceptions, and/or attitudes of the population of interest to be surveyed. Hemera/Thinkstock lan66845_06_c06_p157-190.indd 157 4/20/12 2:48 PM CHAPTER 6 Introduction Introduction I f you’ve ever enjoyed the task of trying to assem – ble a large jigsaw puzzle, you know that different peo – ple have different strategies. Some people like to assemble the edges first, and then work toward the middle. Others like to use the picture on the box to assemble easily recognizable parts of the puzzle. Some like to find all the corners first and work that way. Assembling a puzzle is a complicated task, and different strategic paths can lead to the same solution. When using surveys and ques- tionnaires—the main topics of this chapter—the same prin – ciple applies: There are many topics to consider, and eventually we will get to them all, but we have to start somewhere. Surveys and questionnaires are similar to jigsaw puzzles in that many pieces come together to form the final picture. Nordic Photos/SuperStock Voices from the Workplace Your name: Jessica F. Your age: 30 Your gender: Female Your primary job title: Survey Research Specialist Your current employer: Society for Human Resource Management, Research Department How long have you been employed in your present position? 15 months What year did you graduate with your bachelor’s degree in psychology? 2000 Describe your major job duties and responsibilities. Produce and manage quantitative and qualitative research on HR topics. Design survey instruments and programs online surveys for fielding. Involved in all aspects of data management including the data collection process and performing data quality control. Designs the analysis plan and conducts the analysis using SPSS statistical software. Produces written technical reports. What elements of your undergraduate training in psychology do you use in your work? Coursework in social psychology research methods—learned and applied the fundamentals of survey research methodology, writing technical research reports, running analyses in SPSS, and conducting background research through literature reviews. I also use the information acquired from my statistics course in my job. Coursework in organizational behavior and I/O (industrial/organizational) psychology (e.g., dealing with conflict resolution, change management, motivation, personality tests, (continued) lan66845_06_c06_p157-190.indd 158 4/20/12 2:48 PM CHAPTER 6 Introduction etc.), that are relevant in the human resource profession. Volunteer work as a research assistant in the department of psychology. Spent a year coding data on an emotional experiences study. What do you like most about your job? Meaningfulness of the research—produce research that HR (human resources) professionals and other customers can utilize and apply in their organizations to improve workforce dynamics and make strategic business decisions. Other things that I like about my job include variety of work, managing research projects from beginning to end, the ability to work independently and autonomously. What do you like least about your job? It can be very tedious at times (e.g., data entry, data cleaning, writing) since a high level of accuracy is necessary. The environment is also very structured (e.g., specific procedures and protocols to follow); however, this can vary from job to job. Beyond your bachelor’s degree, what additional education and/or specialized training have you received? I took several classes through SPSS—survey methodology, survey analysis, statistical analysis, syntax, and intermediate topics in SPSS. To design/program web-based surveys—experience in HTML, Dream- weaver, ColdFusion and Microsoft Access. I took classes in most of these areas, however I picked up most of my experience on the job. I have also taken various HR workshops/seminars to stay current with HR and broaden my knowledge base. What is the compensation package for an entry-level position in your occupation? A research assistant position in a non-profit organization in the Washington D.C. area: $22,000–26,000. What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? Medical, dental and vision insurance, 401K, flexible work schedules (e.g., telecommuting, compressed workweek), tuition assistance, professional development opportunities and casual dress. What are the key skills necessary for you to succeed in your career? Ability to pick things up quickly (e.g., learning programming skills, learn about a new topic), strong oral and communication skills, research skills, analytical and problem solving skills, attention to detail and computer skills. I have been fortunate to progress as far as I have in research in the non-profit sector with a bachelor’s degree; however, I do think that at some point in time I will need to get a masters or a doctorate degree. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? Statistics, psychology research methodology class, I/O psychology, and organizational behavior. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? I believe that my research assistantship helped me to get my first professional research position. It made a difference to have real world research experience outside of the classroom. As an undergraduate, do you wish you had done anything differently? If so, what? I wish that I would have joined Psi Chi so that I would have been more active in psychology. I think that it would have helped me to learn more about the field and take advantage of opportunities (e.g., pub – lishing research, presenting, serving on committees, etc.). What advice would you give to someone who was thinking about entering the field you are in? A bachelor’s degree in psychology provides the fundamentals to be successful in just about any line of work. I think that it’s important to try out different types of jobs to see what is a good fit before making a decision to go back to school. A masters or doctorate in psychology is not Voices from the Workplace (continued) (continued) lan66845_06_c06_p157-190.indd 159 4/20/12 2:48 PM CHAPTER 6 Section 6.1 Sampling the Population 6.1 Sampling the Population T he ultimate goal of sampling the population is so that a representative portion of the population can be studied. Thus, by studying the sample carefully and methodi – cally, generalizations can be drawn about the variables or behaviors of interest in the greater population. Two major types of sampling approaches exist— probability sam- pling and non-probability sampling. Why sample? If the goal is to understand how the population thinks, acts, feels, believes, and so on, then why not study the entire popula – tion? First, we often do not have comprehensive lists of members of a population. Say, for example, you wanted to survey all the citizens of Indiana. Is there a comprehensive list of all citizens available? The tax rolls might be a good start, but names and addresses are unlikely to be part of the public record. Plus, some Indiana residents may have moved, or others moved to Indiana. So it is unlikely to have a complete roster of all citizens that is accurate. You can make the same generalization about the students at your college or university, all the individuals in the community with Alzheimer ’s disease, or a list of all the skateboarders in your town. Having an accurate roster of all the members of the popu – lation of interest would be unlikely. In addition, there are other methodological issues as well. Because of the mathematics and probability behind sampling theory, very good samples can be drawn from populations with relatively small margins of error. Dillman, Smyth, and Christian (2009) offer this exam – ple: “one can estimate within ± 3 percentage points the percentage of people who have a high school education in a small county of 25,000 adults with 1,024 completes [completed surveys] and can measure the same thing among the entire U.S. population of more than 300 million by obtaining only 43 more completes” (p. 59). Sampling is efficient. Lastly, survey – ing an entire population might lead to a greater number of non-respondents, and survey researchers become concerned about non-respondents because if bias is driving a person’s choice to not complete the survey, that may weaken the validity of the data (Dillman et al., 2009). We are better suited to select a sampling procedure that allows us to estimate any potential of sampling error in order to obtain a representative sample while minimizing bias and high non-response rates. Probability sampling strives to achieve each of those goals. Probability Sampling There are a variety of approaches to probability sampling, including simple random sam – pling, systematic sampling, stratified sampling, cluster sampling, and multistage sampling. always necessary, and it really depends on what you want to do in the long run. I started out as a research assistant and worked hard and proved that I was capable of doing more. I was promoted twice within about three years. Copyright © 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. Voices from the Workplace (continued) lan66845_06_c06_p157-190.indd 160 4/20/12 2:48 PM CHAPTER 6 Section 6.1 Sampling the Population Each is briefly described in this section. Remember, the overarching goal of probability sampling is that the sample drawn will be representative of the population if all the mem – bers of that population have an equal probability of being selected for the sample. Often, you’ll hear this stated as a non-zero probability (StatPac, 2009; StatTrek, 2009), meaning that there is a chance for every person to be selected, no matter how slim that chance might be. Simple Random Sampling The simple random sample is perhaps the purest form of sampling, and probably one of the rarest techniques used. If you had the roster of the entire population available, you could assign numbers to all members in sample frame, assign random numbers to the possible participants, and then select the sample through a random number table (Babbie, 1973). Random number tables are often found at the back of statistics textbooks just for this purpose. Think of it this way—if we could throw all the names into a large hat and draw a certain target percentage for our survey, in this situation everybody in the survey population has the same probability of being tested (Edwards & Thomas, 1993). Systematic Random Sampling Simply put, in a systematic random sample , every nth person from a list is selected (Edwards & Thomas, 1993). Let’s say that at your college there are 2,000 students currently enrolled, and you determine that you would like to have 100 students complete your sur – vey. Each student completing your survey would have an equal chance of being selected; that is, the probability of being selected is n/N (Lohr, 2008), or in our example, 100/2,000, or 1 out of every 20 students. So, every 20th student would be selected. After determining a random starting point (let’s say No. 4, for example), every 20th student on the roster is selected, meaning the 4th, 24th, 44th, 64th, 84th, 104th, 124th, and so forth (Chromy, 2006). Stratified Sampling Stratified sampling involves an approach where extra precautions are taken to ensure rep – resentativeness of the sample. Strata define groups of people who share at least one com – mon characteristic that is relevant to the topic of the study (StatPac, 2009). The term strata is the plural of stratum; a study can have one stratum, or multiple strata. For example, if you want to ensure that your sample is representative based on gender, then you would stratify on gender. If you know that 55% of the population consists of females and 45% of the pop – ulation consists of males, then you could use random sampling within a gender stratum to extract a sample that matches the gender breakdown of the population precisely. Some – times oversampling is used to decrease sampling error from relatively small groups—that is, researchers may choose to oversample from groups less likely to respond (Edwards & Thomas, 1993). If the percentages in the population match the sample strata selected (as in the gender example above), this is proportionate stratification; if oversampling is used, this practice would be considered disproportionate stratification (Henry, 1990). Cluster Sampling Let’s say you were interested in studying the perceptions of high school seniors enrolled in Advanced Placement (AP) psychology courses throughout the state of New York. It would be difficult to obtain a comprehensive roster of all students at all schools enrolled in AP psy – chology courses. The concept of clustering means that rather than randomize on the level of the individual person, you would randomize on the level of the school where AP psychology lan66845_06_c06_p157-190.indd 161 4/20/12 2:48 PM CHAPTER 6 Section 6.1 Sampling the Population is taught. That is, each “participant” is a school, not an individual person. It probably would be possible to obtain a list of all the schools in New York that offer AP psychology; once the students are assigned to a group or cluster, then the entire cluster is selected or not selected at random (Edwards & Thomas, 1993). One of the general guidelines about cluster sampling is that the researcher desires “to have a larger number of small clustering units than to have a small number of larger clustering units” (Fife-Schaw, 2000, p. 97). The cluster sample tech – nique is particularly useful when it is impossible or impractical to compile an exhaustive list of members composing the target population (Babbie, 1973; Henry, 1990). Multistage Sampling Multistage sampling describes a process that follows after cluster sampling has been implemented. In our AP psychology example, a random sample of New York high schools that offer AP psychology (clusters) is selected for further study. Multistage sampling kicks in once the schools to be studied are selected. For instance, is every high school senior within the selected school/cluster surveyed, or is a systematic random sample drawn? In essence, the multistage sampling approach is two-stage sampling, involving (a) the selection of clusters as a primary selection, and (b) sampling members from the selected clusters to produce the final sample (Chromy, 2006; Henry, 1990). Nonprobability Sampling Nonprobability methods of sampling mean just that; it is unknown what the probability is of each possible participant in the population being selected for the study. Unfortunately, with nonprobability sampling , sampling error cannot be estimated (StatPac, 2009). Two key advantages to nonprobability sampling, however, are cost and convenience (StatTrek, 2009). The main approaches utilizing the nonprobability sampling approach are conve – nience sampling, quota sampling, snowball sampling, and a volunteer sample. Convenience Sampling Convenience samples are just that—convenient. This tech – nique is often used in explor – atory research where a quick and inexpensive method is used to gather data (StatPac, 2009). Psychologists have long relied on convenience samples; for instance, the use of introduc- tory psychology human subject pools represent a convenience sample approach. Quota Sampling Quota sampling as a nonproba – bility sampling technique is the equivalent of stratified sampling Convenience samples are a quick, low-cost method to gather data from an available population of people. If you wanted to have a convenience sample, where would you go? age fotostock/SuperStock lan66845_06_c06_p157-190.indd 162 4/20/12 2:48 PM CHAPTER 6 Section 6.2 Survey Research Methodologies from the probability sampling world. In stratified sampling, you identify key characteris – tics of interest, and then you sample to ensure that those individuals selected represent the population of interest in a proportional manner. In quota sampling, the researcher also desires the strata of interest, but then recruits individuals (non-randomly) to participate in a study (StatPac, 2009). Thus, quotas are filled with respect to the key characteristics needed for survey participants from the population. Snowball Sampling When using the snowball sample technique, members of the target population of inter – est are asked to recruit other members of the same population to participate in the study. This procedure is often used when there is no roster of members in the population, and those members may be relatively inaccessible, such as illegal drug users, pedophiles, or members of a cult (Fife-Schaw, 2000). Snowball sampling relies on referrals and may be a relatively low-cost sampling procedure (StatPac, 2009), but there is a high probability that the individuals who participate may not be representative of the larger population. Volunteer Sample This is a commonly used method for soliciting survey participation, but often the results are quite limited due to the possible motivational differences between volunteers and non-volunteers. When a popular website posts a survey and invites volunteers to partici – pate, the explanatory and predictive power of the data gathered may be suspect (StatTrek, 2009). It is difficult to make confident generalizations from a sample to a population when nonprobability samples are employed, and even less confidence exists if a volunteer sam- ple is utilized. With one piece of the survey/questionnaire puzzle in place (sampling), the next section presents the major survey research approaches or strategies that are com – monly used. 6.2 Survey Research Methodologies T his section provides an overview of the choices that survey researchers must answer concerning how the data are collected. Interviews In some ways, in-person interviews remain the gold standard in survey research. Inter – views have fewer limitations about the types and length of survey items to be asked, and trained interviewers can use visual aids to assist during the interview (Frey & Oishi, 1995)—for example, the interviewee can see, feel, or taste a product (Creative Research Systems, 2009). Interviews are thought to be one of the best ways to obtain detailed infor – mation from survey participants. With an in-person interview, the interviewer and the participant can build rapport through conversation and eye contact, which might allow for deeper questions to be asked about the topic of interest. The drawbacks of interview – ing include high costs and the reluctance of individuals to take the time to complete an interview (Creative Research Systems, 2009; Frey & Oishi, 1995). In addition to one-on- one interviews that may be pre-arranged, there are also intercept interviews, such as those lan66845_06_c06_p157-190.indd 163 4/20/12 2:48 PM CHAPTER 6 Section 6.2 Survey Research Methodologies you may have seen at a mall, where an interviewer intercepts shoppers and asks them for an interview. The level of intimacy that can be achieved with an in-person interview could also be a drawback for some individuals. There are also group interviews, which some call focus groups, where a group of people are interviewed at the same time. Telephone Research In some ways, a growing reluctance to participate in in-person interviews led to the growth of using the telephone as a modality of conducting survey research (Tuckel & O’Neill, 2002). The use of telephone methodology has increased over time, but faces a number of challenges today. For instance, think about how difficult it can be to reach someone on the phone who is willing to participate—Figure 6.1 (from Kempf & Remington, 2007) illus – trates this challenge. Potentialsubject Telephone No telephone Cell phone only Landline Not at home Screencalls Agree to participate Do not screen calls At home Decline By the time you have agreement from a possible participant in a telephone study, a great deal of screening has already occurred. Source: Kempf and Remington, 2007 Figure 6.1: Example of telephone methodology lan66845_06_c06_p157-190.indd 164 4/20/12 2:48 PM CHAPTER 6 Section 6.2 Survey Research Methodologies Coverage has always been a concern of telephone research as well. That is, the greater percentage of homes with a telephone, the better the survey coverage, and the better the possibility of drawing a representative sample from the population of interest. See the fol – lowing for how telephone coverage in the United States has changed over time (Kempf & Remington, 2007): • In 1920, 65% of households did not have a telephone. • In 1970, 10% of households did not have a telephone. • In 1986, 7–8% of households did not have a telephone. • In 2003, less than 5% of households did not have a telephone. As you can see, coverage is quite good regarding households with a phone, but researchers who rely on telephone surveys as their modality for data collection face many challenges today, such as working within the context of Do Not Call lists. Researchers continue to develop new strategies for improving the efficiency of telephone surveys, such as by using computer-assisted telephone interviewing (CATI) systems, random digit dialing (RDD), and interactive voice response systems (“press 1 if you are . . .”). But the challenges seem to be growing as well. The growth of cell phone usage is changing the face of telephone survey research. And that growth has been explosive—from fewer than 500,000 users in 1985 to 35 million users in 1995, and more than 200 million cell phone users in 2005 (Kempf & Remington, 2007). Answering machines, Caller ID, privacy managers, and call blocking services all add to the increasing challenges of conducting survey research by telephone. Mail Surveys Odds are you’ve received a survey in the mail. Did you complete it? Did you give it to someone else in your household to complete? As you can see, there are challenges to using mailed surveys as your modality of survey data collection. There are advantages and dis – advantages of using a particular approach, as explained by de Leeuw and Hox (2008). The advantages to mail surveys include (a) relatively low cost per survey respondent—mailed surveys can be completed with a relatively small staff; (b) no time pressure on the part of the survey respondent; (c) the mailed survey can include visual stimuli, using different scaling techniques and visual cues for survey completions (such as skip patterns); (d) the potential effect (bias) of the interviewer is removed with a mail survey; (e) participants have greater privacy in responding to a mail survey; and (f) if a good sample frame is available with a mailing list, the benefits of random sampling techniques can be realized. The potential disadvantages to mail surveys include (a) potentially low response rates; (b) limited capabilities for complex questions, and the inability for an interviewer to clar – ify questions being asked; (c) when mail is delivered to a household, there is no guarantee that the person for whom the survey is intended is the person completing the survey; and (d) the turnaround time for receiving mailed survey responses can be long. Internet Surveys Participating in a survey facilitated by the Internet could involve invitations through list- servs, discussion groups, advertisements on search engine pages, email directories, pub – lic membership directories, chat room rosters, guest lists from web pages, and of course individual email solicitations (Cho & LaRose, 1999). Compared with paper and pencil lan66845_06_c06_p157-190.indd 165 4/20/12 2:48 PM CHAPTER 6 Section 6.2 Survey Research Methodologies surveys, online/Internet surveys offer a number of advantages (Beidernikl & Kerschbau – mer, 2007), including easy and inexpensive distribution to large numbers of individuals via email, the participant is guided through the survey by essentially filling out a form (i.e., skip patterns are hidden from view), digital resources (e.g., video clips, sound, ani – mation) can be incorporated into the survey design if necessary, and questions can be “required” to be answered as well as verified instantly (e.g., when asked in what year you were born, if something other than a four digit number is entered, the participant can be instantly prompted to use the correct format and prevented from proceeding until making the correction). A number of survey tools are available to assist in the collection of online survey data. Two of the more population choices are SurveyMonkey (http://www.surveymonkey.com ) and Qualtrics (http://www.qualtrics.com ); others include QuestionPro, Zoomerang, KeySurvey, SurveyGizmo, and SurveyMethods. Many of these online survey websites allow you to create an account for free and use it on a limited basis to design a survey and then collect data with that survey (once you exceed a certain number of surveys or a certain number of responses, then most of these sites will want you to purchase an annual membership). After creating your survey, the software will create a custom URL that you then can email to potential par – ticipants or post on a website. You probably have completed a number of online surveys and are familiar with the types of questions and formats. One of the advantages to online survey software is that you can usually download the outcomes/results directly into an Excel file for later analysis (or other types of files, such as SPSS files). Also, some of the sites can assist with rudimentary data analysis (and creating graphs and charts) without even exporting the data. Two key drawbacks of Internet surveys are issues of coverage and nonresponse (de Leeuw & Hox, 2008). The issue of coverage, that is, who has Internet access and who does not, is sometimes referred to as the digital divide (Suarez-Balcazar, Balcazar, & Taylor-Ritzler, 2009). Coverage is a problem for Internet sur – veys (de Leeuw & Hox, 2008), and Suarez-Bal – cazar et al. (2009) provided some specific exam – ples of the possible drawbacks: (a) individuals from low-income and working-class communi – ties are less likely to have access to the Internet; (b) low-income and working-class, culturally diverse individuals are more likely to have only one computer, which would limit the potential for completing Internet-based surveys; (c) lim – ited access often translates into limited famil – iarity with online/Internet applications, and (d) there may be cultural barriers that make Internet research more difficult to successfully accomplish (more on this in a moment). In addition to the challenge of coverage, there is also the challenge of representativeness . An Internet survey approach may not achieve the level of representativeness desired (Beidernikl The Internet can facilitate many types of surveys, which are easier and less expensive than regular paper and pencil surveys. PR Newswire/Associated Press lan66845_06_c06_p157-190.indd 166 4/20/12 2:48 PM CHAPTER 6 Section 6.3 Comparisons of Methodologies & Kerschbaumer, 2007; de Leeuw & Hox, 2008). In fact, you can think about whether those replying to an Internet survey are representative of the entire population, represen- tative of the Internet population, or even representative of a certain targeted population (Beidernikl & Kerschbaumer, 2007). Add in the complexity of culture, and you can see that well-designed Internet surveys can take a significant amount of work. Consider this example offered by Suarez-Balcazar et al. (2009): For instance, in the Chicago Public Schools, students speak over 100 dif – ferent languages and dialects. Social scientists planning studies in these types of settings must consider how they are going to communicate with the participants’ parents. Although children of first generation immigrants may be able to speak, read, and participate in Internet-based surveys in English, information such as consent forms and research protocols that are sent to the parents may need to be translated into their native language and administered using paper-and-pencil format. (p. 99) If not used carefully, online/Internet survey researchers are capable of invading privacy (Cho & LaRose, 1999), and care should be taken to minimize that threat. 6.3 Comparisons of Methodologies W ith all the different modalities of survey administration, the natural ques – tion arises—which approach is best? The answer to that complex question is it depends . However, there have been some very useful studies conducted that compare the different methodologies, and below is a sampling. de Leeuw and Hox (2008) report that, on average, web-based surveys have an 11% lower response rate than mailed and telephone surveys. In an experiment that directly compared regular mail and e-mail surveys, Schaefer and Dillman (1998) found comparable response rates—57.5% for regular mail, and 58.0% for e-mail. When Braunsberger, Wybenga, and Gates (2007) compared telephone surveys and web-based surveys, a two-wave web- based approach provided more reliable data estimates than telephone surveys, and at a lower cost: Each telephone survey cost $22.75 to complete, whereas the cost of each web-panel survey was $6.50. What does the future hold for preferred survey research modality? In addition to the particularly useful comparison stud – ies, a growing trend is to utilize a mixed- mode approach (e.g., Nicolle & Lou, 2008), where multiple modalities are accessed to achieve the research goals. Thus, you may see email reminders to participate in a tele – phone survey. The mixed-mode approach The mixed-mode approach uses several methods to gather research. What are the benefits of this approach? The drawbacks? iStockphoto/Thinkstock lan66845_06_c06_p157-190.indd 167 4/20/12 2:48 PM CHAPTER 6 Section 6.4 Designs for Survey Research can also involve the collection of qualitative data as well as quantitative data. Qualita – tive data, such as the responses to open-ended questions on a survey (e.g. “How do you feel about parking on your campus?), can provide particularly rich and useful data, and qualitative approaches are often the most helpful when we know the least. In the Nicolle and Lou (2008) example, faculty members were asked about the process by which they adopt new technologies for use in college courses, and some faculty completed surveys, whereas others were interviewed in person—thus, a mixed-mode approach. In another example, McDevitt and Small (2002) used both Internet and mail to survey participants of an annual sporting event. If the sampling plan and survey modality puzzle pieces are in place, another decision to be made is the overall design of the survey research. In some regard, these concepts do overlap with topics from Chapter 8 on quasi-experimental research designs. But a brief review of how these design decisions affect survey research is warranted here. 6.4 Designs for Survey Research A lthough different researchers may use slightly different terminology, the major cat – egories of survey research designs are presented in this section. Cross-Sectional Survey Designs In a cross-sectional survey design , data collection occurs at a single point in time with the population of interest (Fife-Schaw, 2000; Visser, Krosnick, & Lavrakas, 2000). One way to think about a cross-sectional sur – vey is that it is a snapshot in time (Fink & Kosecoff, 1985). Cross- sectional surveys are relatively inexpensive (Fife-Schaw, 2000) and relatively easy to do (Fink & Kosecoff, 1985). However, if the landscape changes rapidly, and that amount of change is impor – tant to your survey research, then using a cross-sectional design will not allow you to capture this change over time (Fife-Schaw, 2000; Fink & Kosecoff, 1985). Longitudinal Survey Designs A longitudinal survey is con – ducted over time, but this label alone does not give us enough details about the type of longitudinal survey. Longitudinal studies face unique challenges, such as keeping track of respondents over time and how Cross-sectional survey design gathers data from the population all at one time, as shown in this call center that collects survey information for clients. Marka/SuperStock lan66845_06_c06_p157-190.indd 168 4/20/12 2:48 PM CHAPTER 6 Section 6.4 Designs for Survey Research to motivate respondents to continue to respond in the future (Dillman et al., 2009). In general, the key advantage of longitudinal designs is that they allow for the study of age- related development. However, this can be confounded with events over time that might influence your variables (Fife-Schaw, 2000). For example, if you are interested in how individuals feel about their personal safety, and the span of your longitudinal research includes September 11, 2001, then your research might be affected by that historical event, and changes may not be due only to the passage of time. Attrition (dropping out of the study over time) is a drawback, and participants repeatedly tested over time can be sus – ceptible to the demand characteristics of the research—having participated multiple times in the past, the participants know what is expected and probably understand the variables and general hypotheses being tested (Fife-Schaw, 2000). Cohort and Panel Survey Designs In a cohort study , new samples of individuals are followed over time, whereas in a panel study, the same people are tracked over time (Jackson & Antonucci, 1994). In a panel study , the same people are studied over time, spanning at least two points in time (Fink & Kosecoff, 1985; Jackson & Antonucci, 1994; Visser et al., 2000). This type of study can be particularly useful for understanding why particular changes are occurring over time, because you are asking the same individuals to respond over time (you also have a base – line comparison measure from when they first entered the study). There are so many more variations of possible research designs, such as trend studies, population sampling, and even an approach called the “multigenerational lineal panel” approach (Jackson & Antonucci, 1994). The key to remember for now is that there are many pieces of this puzzle to be solved, and the survey research design that psychologists select is based on a number of factors. But the types of questions that we can answer are strongly governed by how we ask the question. This is illustrated in the “Classic Studies in Psychology” story that follows, and much of the remainder of this chapter is devoted to providing helpful advice about crafting your own survey questions, selecting the scales of measurement, and choosing data analysis strategies to make the most of survey data. Classic Studies in Psychology: Loftus and Eyewitness Testimony (Loftus & Palmer, 1974; Loftus, 1975) As you will see, psychologist Elizabeth Loftus cleverly studied the relationship between the phrasing of a question and the impact of that phrasing on the answer. Not only is this an important consideration for survey research, but this line of research helped Loftus to develop expertise concerning eyewitness testimony (and how asking questions may lead to the creation of false memories). In the Loftus and Palmer (1974) experiment, 45 stu- dents were shown 7 films being used by a local Seattle Police Department as part of their driver’s education program. Following each film, the participants were asked to write about the film they had just seen and to answer a series of survey questions—the key research question asked about the speed at which the cars were going when the collision Associated Press (continued) lan66845_06_c06_p157-190.indd 169 4/20/12 2:48 PM CHAPTER 6 Section 6.4 Designs for Survey Research occurred. However, for the 45 students who viewed the accident film, groups of nine were asked dif- ferent questions, as presented in the table below. After being asked the particular question (note the key word in boldface), students responded with their average speed estimate of the two cars, in miles per hour (mph). The results are presented in Table 6.1. Table 6.1: Loftus and Palmer survey questions and estimates Survey Question Average Speed Estimate About how fast were the cars going when they smashed each other?40.5 mph About how fast were the cars going when they collided with each other? 39.3 mph About how fast were the cars going when they bumped each other? 38.1 mph About how fast were the cars going when they hit each other? 34.0 mph About how fast were the cars going when they contacted each other?31.8 mph Loftus and Palmer (1974) found these speeds to be significantly different. Thus, even the verb used to ask the question made a significant difference in how memories were reported. But Loftus’ creative thinking about these issues continued. In a study published a year later, Loftus (1975) further explored how survey answers were dependent on the questions, and furthermore, how embedding false information in the original survey questions can lead to the embedding of false memories over time. This classic study reports the outcomes of four different experiments, but we’ll only describe two of those experiments here. In Experiment 1, stu – dents “were shown a film of a multiple-car accident in which one car, after failing to stop at a stop sign, makes a right-hand turn into the main stream of traffic. In an attempt to avoid a collision, the cars in the oncoming traffic stop suddenly and a five-car, bumper-to-bumper collision results. The film lasts less than 1 min., and the accident occurs within a 4-sec. period” (p. 563). The key car in the scenario (Car A) is then presented as a part of a diagram with the other cars. Half the students were asked, “How fast was Car A going when it ran the stop sign?” and the other half of students were asked, “How fast was Car A going when it turned right?” However, in this study, the key question of interest was not about miles per hour but rather is “Did you see a stop sign for Car A?” See the results in Table 6.2. Table 6.2: Survey results Leading Question Answer to the Next Question “Did you see a stop sign for Car A?” How fast was Car A going when it ran the stop sign? 53% answer YES How fast was Car A going when it turned right? 35% answer YES Just mentioning the stop sign in the question helps participants remember that there was a stop sign. But what if leading questions contained misinformation? What impact would that have on memory? Loftus addressed that issue in Experiment No. 4 in her 1975 study. She showed students a 3-minute film of an automobile that eventually collides with man pushing a baby carriage. After viewing Classic Studies in Psychology: Loftus and Eyewitness Testimony (Loftus & Palmer, 1974; Loftus, 1975) (continued) (continued) lan66845_06_c06_p157-190.indd 170 4/20/12 2:48 PM CHAPTER 6 Section 6.4 Designs for Survey Research the film, the participants are asked 45 questions about the film, but Loftus is only interested in 5 of the answers. In the “Direct” condition, the participants were asked a straightforward question, such as, “Did you see a woman pushing the carriage?” (We know from the description above that the correct answer is no.) In the “False Presupposition” condition, participants were asked, “Did the woman push- ing the carriage cross into the road?” A third group served as the control group and did not receive any key questions at all (just filler questions). One week later, the participants returned and were asked the direct question—in this case, did you see a woman pushing the carriage? See Table 6.3 to find out what happens one week later. Table 6.3: Experiment No. 4 follow-up questions and results Experimental Condition Percentage YES Responses to “Did you see a woman pushing the carriage?” Direct—Did you see a woman pushing the carriage? 36% YES False Presupposition—Did the woman who was pushing the carriage cross into the road? 54% YES Control (No leading question) 26% YES Note: Remember, it was a man who was pushing the carriage in the film. If memory were working perfectly, the percentage of YES in all three rows should be 0%. Note that for the control group, without any leading questions at all, 26% remember a woman push – ing the carriage, when in fact it was a man. But look what happens one week later—the amount of misremembering increases, and it can be manipulated by the researcher. You should know that Loftus did this with other scenarios throughout the study (1975), as well as in other studies (e.g., Loftus & Hoffman, 1989). These fascinating outcomes have continued to influence Loftus’ work, and have influ – enced the work of others as well (e.g., Crombag, Wagenaar, & van Koppen, 1996). If you think about it, the ability to change memories based on the way that a question is asked has important implications for issues such as eyewitness testimony and repressed memories, two top – ics that Loftus has explored throughout her career. Niland (2007) correctly pointed out that Loftus’ research squarely puts her in the center of the controversy about repressed childhood memories, and that it is possible to implant a false memory. This capability (or an accusation to some) threatens a number of therapists and victims of abuse who have come to believe that the memories of the abuse have been repressed for years, and with the help of a psychotherapist those memories can be discov – ered (Niland, 2007). Elizabeth Loftus has received many accolades for her work about the formation and manipulation of memory. When Philip Zimbardo (President of the American Psychological Association in 2002) wrote about “does psychology make a significant difference in our lives,” Loftus’ research was listed as semi- nal work in the area of eyewitness identification (Zimbardo, 2004). In 2004, Loftus was elected to the National Academy of Sciences (a high honor); she was also named as one of the 100 most influential psychologists of the 20th century and the highest ranked woman on the list (Zagorski, 2005). Reflection Questions 1. How does the careful selection of the verb used in the experiments by Loftus compare to the types of verbs you might select to develop survey research questions for a project at work? Is there a chance that the specific wording selected might have an impact on the results you observe? Why or why not? Classic Studies in Psychology: Loftus and Eyewitness Testimony (Loftus & Palmer, 1974; Loftus, 1975) (continued) (continued) lan66845_06_c06_p157-190.indd 171 4/20/12 2:48 PM CHAPTER 6 Section 6.5 Scaling Methods 6.5 Scaling Methods A s you can surely see by now, survey research is a complex puzzle with multiple pieces needing to be put into place before the picture is complete. Perhaps one of the most complicated parts of survey research is deciding on the scale by which to measure a person’s attitudes, opinions, behavior, knowledge, etc.—in fact, there are entire books on the subject (e.g., Netemeyer, Bearden, & Sharma, 2003). As you read ear – lier in Loftus’ work, how you ask the questions does shape the answer you receive. In fact, how you shape the possible answers can even influence the answers you receive. For example, Schwartz (1999) reported on some of his previous research where he had sur – veyed German respondents about the number of hours per day that they watch television. Two groups were asked the same question but given different response categories—these response categories are depicted in Table 6.4. Table 6.4: How response scales can shape the results—daily TV consumption Low Frequency AlternativesPercent Reporting High Frequency AlternativesPercent Reporting Up to ½ hour 7.4% ½ hour to 1 hour 17.7% 1 hour to 1 ½ hours 26.5% 1 ½ hours to 2 hours 14.7% 2 hours to 2 ½ hours 17.7%Up to 2 ½ hours62.5% More than 2 ½ hours 16.2%2 ½ hours to 3 hours23.4% 3 hours to 3 ½ hours 7.8% 3 ½ hours to 4 hours 4.7% 4 to 4 ½ hours 1.6% More than 4 ½ hours 0.0% 2. Have you ever been in a car accident or spoken to someone who has? Think about your memory for that event (or ask the person about his or her memory for that event). Is the memory like a flashbulb memory, where every element of the scene is remembered, or have some memories faded over time while other “memories” seem to have been invented? What about the effect of an emotional reaction during a car accident, such as the rush of adrenaline in anticipation of the fight-or-flight response? How do these individual factors need to be considered and combined to better our understanding of memory for these kinds of events? 3. Eyewitness testimony has important ramifications for how our criminal justice system works. Eyewitness testimony can help clear some people of crimes, whereas eyewitness testimony some – times provides key evidence that leads to the incarceration of an individual. Given the fallibility of memory, does the legal system have checks and balances in place to help prevent misremember – ing and to minimize the fallibility of eyewitness testimony? Classic Studies in Psychology: Loftus and Eyewitness Testimony (Loftus & Palmer, 1974; Loftus, 1975) (continued) lan66845_06_c06_p157-190.indd 172 4/20/12 2:48 PM CHAPTER 6 Section 6.5 Scaling Methods Look what happens, depending on the response scale. When the scale starts low (left side of table), only 16.2% of respondents report watching more than 2 ½ hours of television per day, but when the alternatives start higher on the scale (on the right side of the table), 37.5% of respondents report watching more than 2 ½ hours of television per day. Just by the scale difference alone, the magnitude of this difference makes it difficult to draw meaningful conclusions. So what do we do about situations where we need to design sur – veys and items and scales? We rely on best practices and established research that guides the decision making necessary to select an appropriate scale. What follows is a brief over – view of the major types of scales you are likely to use. Dichotomous Scales When you use a dichotomous scale , there are only two possible options. So if the possible options are agree/disagree, yes/no, true/false, male/female, and so on, then you are using a binary scale. Respondents provide nominal scale data (this is an important consideration for later data analysis options). Some examples of dichotomous scales where a yes/no type of response would be ade – quate are: • I am married. • I download music illegally. • My parents are divorced. Some argue (e.g., Spector, 1992) that single yes/no questions are insufficient, because they are not sen – sitive to subtle change over time, they dictate that individuals place themselves into large categories, and that many psychological phenomena are so complex that a singular yes/no response may fail to capture the complexity. As you design your sur – veys, keep in mind that the hypotheses you wish to test will help to inform you if a dichotomous scale can yield the type of information you seek. Likert Scales Likert scales , or perhaps Likert-type scales, may be the most famous/popular type of scale used by psychological researchers today. The Likert scale is named after the psy – chologist from the University of Michigan, Rensis Likert (pronounced Lick-ert). Likert’s seminal work (1932), now called a Likert scale, called for a survey response scale to have a 5-point scale, measuring from one pole of disagreement to the other pole of agreement. Each of the scale points has a specific verbal description (Wuensch, 2005). A declarative statement is made, and then the respondent selects the appropriate answer. The low value is strongly disagree, and the high value is strongly agree, like this: When using a dichotomous scale, there are only two possible options, such as yes or no. Mauritius/SuperStock lan66845_06_c06_p157-190.indd 173 4/20/12 2:49 PM CHAPTER 6 Section 6.5 Scaling Methods 1 = strongly disagree 2 = disagree 3 = neutral (neither agree nor disagree) 4 = agree 5 = strongly agree There have been many variations and changes suggested that are loosely based on the above criteria, so you will often see “Likert-type” scale used rather than the very spe – cific Likert scale as described above. For example, Fowler (1988) has made the argument that Likert-type variations (shown below) might be better suited because they would have lesser emotional ties: 4 = completely agree, 3 = generally agree , 2 = generally disagree, and 1 = completely disagree or 4 = completely true, 3 = mostly true , 2 = mostly untrue, and 1 = completely untrue . Of course, these would not conform to the true Likert scale but would be categorized as Likert-type scales. There have been many variations on this theme. The following examples demonstrate many of these variations, as presented by Vagias (2006). Note the varying types of response anchors possible with a Likert-type scale approach, including the use of frequency, truthfulness, probability, importance, concern, support, usage, awareness, satisfaction, and influence. As you think about the type of scale you might employ in your survey research, and you examine the following examples, you should begin to appreciate just how useful and versatile using a Likert-type scale can be. Level of Acceptability 1 – Totally unacceptable 2 – Unacceptable 3 – Slightly unacceptable 4 – Neutral 5 – Slightly acceptable 6 – Acceptable 7 – Perfectly Acceptable Level of Importance 1 – Not at all important 2 – Low importance 3 – Slightly important 4 – Neutral 5 – Moderately important 6 – Very important 7 – Extremely important Knowledge of Action 1 – Never true 2 – Rarely true 3 – Sometimes but infrequently true 4 – Neutral 5 – Sometimes true 6 – Usually true 7 – Always true Level of Problem 1 – Not at all a problem 2 – Minor problem 3 – Moderate problem 4 – Serious problem Level of Awareness 1 – Not at all aware 2 – Slightly aware 3 – Somewhat aware 4 – Moderately aware 5 – Extremely aware Likelihood 1 – Extremely unlikely 2 – Unlikely 3 – Neutral 4 – Likely 5 – Extremely likely Level of Satisfaction – 5 point 1 – Very dissatisfied 2 – Dissatisfied 3 – Unsure 4 – Satisfied 5 – Very satisfied Level of Appropriateness 1 – Absolutely inappropriate 2 – Inappropriate 3 – Slightly inappropriate 4 – Neutral 5 – Slightly appropriate 6 – Appropriate 7 – Absolutely appropriate (continued) Likert-Type Scale Response Anchors lan66845_06_c06_p157-190.indd 174 4/20/12 2:49 PM CHAPTER 6 Section 6.5 Scaling Methods Level of Agreement 1 – Strongly disagree 2 – Disagree 3 – Somewhat disagree 4 – Neither agree or disagree 5 – Somewhat agree 6 – Agree 7 – Strongly agree Frequency – 5 point 1 – Never 2 – Rarely 3 – Sometimes 4 – Often 5 – AlwaysLevel of Familiarity 1 – Not at all familiar 2 – Slightly familiar 3 – Somewhat familiar 4 – Moderately familiar 5 – Extremely familiar Level of Difficulty 1 – Very difficult 2 – Difficult 3 – Neutral 4 – Easy 5 – Very easy Level of Quality – 5 point 1 – Poor 2 – Fair 3 – Good 4 – Very good 5 – Excellent Level of Satisfaction – 5 point 1 – Not at all satisfied 2 – Slightly satisfied 3 – Moderately satisfied 4 – Very satisfied 5 – Extremely satisfied Source: Vagias (2006). Likert-Type Scale Response Anchors (continued) Thurstone Scale and Guttman Scale Both the Thurstone scale and the Guttman scale describe a methodology of scale develop – ment as well as measuring individual responses. In 1928, Thurstone proposed the technique (now called the Thurstone scale) to develop a response scale of equally appearing intervals by having participants make a series of comparative judgments (Page-Bucci, 2003; Roberts, Laughlin, & Wedell, 1999). First, a large number of attitude statements would presumably represent the entire range of possible options, and respondents would provide a global eval – uation of favorability or unfavorability toward the topic presented in the survey items—for instance, a pairwise comparison could be presented, where a respondent is forced to choose which statement he or she agrees with more, and the process is repeated over and over. From a group of individuals, this yields a hierarchy of agreement scores for each item, and then in the second stage individuals re-rate the items in terms of agreement or disagreement (Page-Bucci, 2003; Roberts et al., 1999). The goal of using this multistage process is so that the final items retained in the survey fit the respondents’ patterns of answering well, rather than hoping that survey items capture what the respondents think about a particular topic. A Guttman scale is difficult to construct because it is based on generating a set of items that increase in difficulty; on a 7-item scale, if the easiest item to agree to is Item No. 1, and the most difficult item to agree to is Item No. 7, and you agree with Item No. 5, that automatically means that you agree with the first four items as well. In other words, what – ever item you agree with on the hierarchy, it is assumed that you agree with all the items leading up to it also. Page-Bucci indicated (2003) that although this scale may allow for more complex measures than a Likert-type scale, the scales are difficult to construct and the scoring systems are cumbersome. Semantic Differential Scales The semantic differential scale technique, developed by Osgood in the 1950s, is a scale that is designed to measure affect or emotion (Henerson, Morris, & Fitz-Gibbon, 1987), but lan66845_06_c06_p157-190.indd 175 4/20/12 2:49 PM CHAPTER 6 Section 6.5 Scaling Methods it can measure much more than that. Using adjectives that are polar opposites, participants are asked to select how they feel about the survey topic being presented. For example, to respond to the question “Thinking about this course, how do you feel about the grading policies being used?” the surveyed person would be asked to place on a checkmark on one of the seven lines spanning the polar opposites on the the semantic differential scale below: fair ___ ___ ___ ___ ___ ___ ___ unfair unreliable ___ ___ ___ ___ ___ ___ ___ reliable confusing ___ ___ ___ ___ ___ ___ ___ clear helpful ___ ___ ___ ___ ___ ___ ___ not helpful good ___ ___ ___ ___ ___ ___ ___ bad Based on prior research, three types of findings tend to emerge from the use of semantic differential scales (Page-Bucci, 2003): an evaluative fac – tor (good-bad), an intensity/potency factor (strong-weak), and an activity factor (slow-fast). Responses on these items can be given a score of 1 to 7, depending on where the mark on the scale occurred; most researchers ana – lyze these data the same as they would Likert-type agreement scale data—as interval/ratio (scale) data. The seman – tic differential scale is good at captur – ing feelings and emotions, is relatively simple to construct, and is relatively easy for participants to use, but the resulting analyses can be complicated (Page-Bucci, 2003). An example of more possible pairings appears below (from Henerson et al., 1987): The use of semantic differential scales reveals three types of findings: evaluative; intensity/potency; and activity. This type of scale is helpful in recording feelings and emotions. iStockphoto/Thinkstock angry-calm bad-good biased-objective boring-interesting closed-open cold-warm confusing-clear dirty-clean dull-lively dull-sharp irrelevant-relevant last-first lan66845_06_c06_p157-190.indd 176 4/20/12 2:49 PM CHAPTER 6 Section 6.5 Scaling Methods not brave-brave old-new passive-active purposeless-purposeful sad-funny slow-fast sour-sweet static-dynamic superficial-profound tense-relaxedugly-pretty unfair-fair unfriendly-friendly unhappy-happy unhealthy-healthy uninformative-informative useless-useful weak-strong worthless-valuable wrong-right Other Types of Scales There are many more types of scales that are used in survey research. Visual analog scales can be used to obtain a score along a continuum, where a participant places a checkmark to indicate where his or her attitude or opinion falls along the scale. Below is an example of the visual analog scale: No pain at all ———————————— The worst pain I ever experienced This would be an example of a subjective continuum scale, where a checkmark is made along the scale to indicate how positive or negative a respondent’s opinion is about a particular topic: Very positive ———————————————————— Very negative With the advent of online survey packages, the visual analog scale has become digital. In the online survey software package Qualtrics, visual analog scales are presented as “slid – ers,” and respondents can click on the pointer and slide it to location along the continuum that represents their belief. See Figure 6.2 for an example of a series of slider questions. lan66845_06_c06_p157-190.indd 177 4/20/12 2:49 PM CHAPTER 6 Section 6.5 Scaling Methods Completely dissatisfied 01 0203 04050607 0809 0 100 My co-workers The workplace environment My company in general My direct supervisor My annual compensation The opportunities for advancement Completely satisfied Please rate your overall level of SATISFACTION for each of the workplace categories below. Move the slider to the appropriate level: 0 = completely dissatis fied and 100 = completely satisfied. This is an example of a visual analog scale used in survey research in a survey software program called Qualtrics. Participants click on the blue arrow and drag it to the location that indicates their answer. Source: Qualtrics, 2011 Figure 6.2: Example of a visual analog scale Surveys do have advantages though: They allow for anonymity of responses and sta – tistical analysis of large amounts of data, they can be relatively cost effective, sampling mechanisms can be carefully controlled in some cases, and by using standardized ques – tions change can be detected over time (Seashore, 1987). Some of the limitations and risks of the survey research approach include a lack of control over variables of interest, response rates may be problematical, ambiguous surveys may lead to difficult interpre – tation, in some contexts participants may not believe their data are truly anonymous and confidential, the possibilities of bias due to non-response or socially desirable responding, and the inability to draw cause-and-effect conclusions (Fowler, 1998; Sea – shore, 1987). Surveys are pervasive in psychology and throughout culture. The ability to properly design a survey and interpret its results appropriately is a skill that well-suits psychology majors for a future in the workplace, or for graduate school first and then the workplace. But it is important to remember that surveys are a measure of self-report and not actual behavior. There are multiple reasons why survey data may be inaccurate; it could be that the respondents don’t know the answer, know the answer but can’t recall the answer, don’t understand the question (but answer anyway), or just choose not to answer for whatever reason (Fowler, 1998). Because most survey research does not share the same lan66845_06_c06_p157-190.indd 178 4/20/12 2:49 PM CHAPTER 6 Section 6.6 Analysis of Survey Data characteristics as experimental designs, it is important not to over-interpret the results of survey research. The survey approach is powerful in helping psychologists identify the relationships between variables and differences among groups of people, but the results are only as good as the design quality that is necessary for this complex task. 6.6 Analysis of Survey Data I n most respects, analyzing survey data is the same as analyzing any other type of data—your analysis choices are based on your hypotheses, the scales of measurement, the tools available for data analysis, and so on. Before mentioning specific approaches for data analysis, let’s review at a conceptual level the types of errors that are encountered in survey research. Remember that errors in this context are not mistakes but are the pos – sible outcomes of the study that the researcher cannot account for—that is, the changes or values of the dependent variable that are not due to the independent variables being manipulated, controlled, or arranged. Types of Errors In classic psychometric measurement theory, the total amount of error is assumed to be the sum of measurement error + sampling error (Dutka & Frankel, 1993). Those who study survey research design further categorize the types of threats and errors that can occur with this type of research. Although Dillman et al. (2009) were referring specifically to Internet panel research in this case, they present a four cornerstone model of surveying and errors that is useful here for our greater understanding. A coverage error in survey research refers to the methodology used. For example, if an Internet approach is used, only about 70% of households have Internet access, so cover – age error exists (Dillman et al., 2009). The coverage error is much smaller with telephone surveys, but the proportion of individuals with landlines is decreasing whereas cell phone subscribers are increasing (Kempf & Remington, 2007). Survey researchers need to be cog – nizant of coverage error concerns when making methodological choices. A sampling error occurs when not all the potential participants from a population are rep – resented in a sample (Dutka & Frankel, 1993), and this is often due to the sampling method utilized by the researcher (Futrell, 1994). In fact, this sampling procedure is so important that it was the opening puzzle piece of this chapter. Another related sampling issue is volunteerism , or self-selection. When a study relies on volunteers (for whatever reason), there is always a concern that volunteers may behave differently than non-volunteers, and if this is the case, it weakens the generalizability of the survey results. In fact, Rosenthal and Rosnow (1975) have reliably demonstrated that volunteers differ from non-volunteers in the following ways: (a) volunteers are more educated than non-volunteers; (b) volun – teers are from a higher social class than non-volunteers; (c) volunteers are more intelligent than non-volunteers; (d) volunteers are more approval-motivated than non-volunteers; and (e) volunteers are more sociable than non-volunteers. However, if the only way you can conduct your research is through volunteers, then that is what you do. But it would be important to remember these caveats when drawing conclusions from your survey lan66845_06_c06_p157-190.indd 179 4/20/12 2:49 PM CHAPTER 6 Section 6.6 Analysis of Survey Data research (or any research) that depends exclusively on volun- teer participants. Measurement error can occur due to a number of reasons, but measurement errors tend to fall into the category of mea – surement variation (the lack of a reliable instrument) and measurement bias (asking the wrong questions, or using the results inappropriately) (Dutka & Frankel, 1993). As in any com – plex enterprise, the potential for mistakes can be high, and Futrell (1994) listed some com – mon measurement errors that can occur in survey research: 1. Failing to assess the reliability of the survey. 2. Ignoring the subjectivity of participant responses in survey research. 3. Asking non-specific survey questions. 4. Failing to ask enough questions to capture the behavior, opinion, or attitude of interest. 5. Utilizing incorrect or incomplete data analysis methods. 6. Drawing generalizations that are not supported by the data nor the data analysis strategy selected. Essentially, measurement errors address issues of (a) did we measure what we thought we measured, and (b) did we interpret the results appropriately? Non-response error is of particular concern in survey research (Dillman et al., 2009). As a general rule, if there is a response rate of 25% or less (or a non-response rate of 75% or more), then the survey researcher should be concerned with the question “Are those responding to my survey different from those not responding to my survey?” (Dillman et al., 2009). There are many different approaches for dealing with high non-response rates, and some of those methods involve weighting the responses that are received (Dale, 2006) as well as specifically following up with a subset of non-responders and asking them why they didn’t respond. The goal here would be to determine that there was no systematic bias in why people responded or did not respond to the initial survey request. If there is no bias (that is, no systematic reason driving non-response), then the non-response rate is less of a concern to the survey researcher. Data Handling Issues The details and complexity of data handling issues within survey research are beyond the scope of this chapter, but two issues are worth mentioning, if only briefly. After collecting When volunteers are used in sampling, there is concern that volunteers could change the survey results by behaving differently than non-volunteers. How might this be addressed? iStockphoto/Thinkstock lan66845_06_c06_p157-190.indd 180 4/20/12 2:49 PM CHAPTER 6 Section 6.6 Analysis of Survey Data your data, but prior to analysis, you will have to do some data “cleaning” (sometimes called data editing). Even though every survey researcher must do this, there are not com – monly accepted standards for data cleaning (Leahey, Entwisle, & Einaudi, 2003). Some – times it involves the elimination of outliers (which is relatively straightforward), but other times data decisions are more complex. For example, someone may be hand-coding data into an SPSS file, and on a written survey form completed by a college student, the student filled in “Age: ____” with 107. It would be pretty clear from this scenario that there was not a 107-year-old college student in the laboratory setting when the data were collected, so this value should be discarded from “age” variable (thus, this participant has missing data). But this brings other issues to mind: If this respondent reports many outliers, did he or she take the survey seriously? Should just the age value be discarded, or should the entirety of the survey responses from this individual be deleted? Data cleaning decisions can become more complex. Let’s say you are asking survey items where the responses are made on a Likert-type agreement scale, where 1 = strongly dis- agree , 2 = disagree, 3 = neutral , 4 = agree , and 5 = strongly agree . One coded response to the statement “I am comfortable with the undergraduate major I have selected,” is 55. What do you do? Do you assume the respondent meant a 5 (strongly agree ), and change the response? Is it possible to go back and confirm what the participant meant, or were the data collected anonymously? You could guess that a 55 meant a 5, but what about a 23 entry? Did the person mean 2 (disagree ) or 3 (neutral )? Here’s one more: In an online sur – vey, where respondents directly enter their age, a participant enters the value 1.9. Should that be recoded as 19 years old, or should the data be deleted? These data cleaning issues are also related to how survey researchers handle missing data, and there are a number of complex approaches for that (Dale, 2006; Graham, Taylor, Olchowski, & Cumsille, 2006; Rudas, 2005). As a psychologist/survey researcher-in-train – ing, you should err on the side of caution. If you cannot confirm what a participant meant by his or her response, delete it. As you become more savvy at performing data cleaning and missing data analyses, you can alter this conservative approach. Furthermore, if you collect your survey data anonymously, you have no method of contacting individuals to clarify their intended response. We’ll discuss more data cleaning issues in Chapter 7. Data Analysis Approaches As alluded to earlier, the possibilities for analyzing survey data are vast, and they depend on many of the same characteristics of other data analysis situations, such as the scale of measurement, the amount of data available, and the hypotheses to be tested. It would not be possible to summarize all of the options here, as entire books are available about the subject (Fink, 1995). Data analytic strategies can become more or less complicated, how – ever. If your goal is to communicate effectively with the public, you might not choose to present the results of a repeated measures ANOVA, but you might present a table of means or a bar graph that clearly and succinctly communicates the story you want to tell. If you are comparing two nominal scale variables, such as gender differences on how respon – dents answered a categorical survey item (“Are you married?”), then a chi-square analysis would be appropriate. Essentially, you will need the knowledge that you (hopefully) learn from a statistics course to be able to analyze your survey data. This is why some call the Statistics-Research Methods sequence the core of the undergraduate psychology major. lan66845_06_c06_p157-190.indd 181 4/20/12 2:49 PM CHAPTER 6 Section 6.6 Analysis of Survey Data Data analyses can range from simple to complex. Table 6.5 is an example of “complex,” as Roelen, Koopmans, and Groothoff (2008) reported the predictors between overall job satisfaction and specific aspects of a job. These researchers used survey research as their method of data collection and a multiple regression as part of their data analysis strategy. Table 6.5: Correlation between overall job satisfaction and specific job aspects Mean (SD)B (SE) Age (years) 38.1 (10.3)0.01 (0.00)0.05 Gender (female = 0, male = 1) −0.11 (0.09)−0.04 Educational level 1.7 (0.7) Primary education relative to tertiary 0.15 (0.14)0.06 Secondary education relative to tertiary 0.18 (0.13)0.07 Physical demands (range 17) 4.0 (1.9)0.02 (0.02)0.03 Psychological demands (range 1–7) 4.1 (1.7) −0.04 (0.03) −0.05 Job autonomy (range 1–7) 5.4 (1.4)0.09 (0.04)0.09* Decision latitude (range 1–7) 4.8 (1.7) −0.01 (0.03) −0.02 Career perspectives (range 1–7) 4.2 (1.7)0.12 (0.04)0.16* Overall satisfaction (range 1–7) 5.3 (1.3) Specific satisfaction (range 1–7) with: Colleagues 5.6 (1.2)0.15 (0.04)0.14 ** Work times 5.5 (1.4) −0.02 (0.04) −0.03 Task variety 5.1 (1.5)0.28 (0.04)0.31** Supervisor 4.7 (1.7)0.06 (0.04)0.07 Working conditions 4.7 (1.5)0.11 (0.04)0.13** Workload 4.7 (1.4)0.11 (0.05)0.12* Work pace 4.7 (1.5)0.02 (0.04)0.02 Salary 4.3 (1.6) −0.05 (0.03) −0.06 Work briefings 4.3 (1.8) −0.01 (0.04) −0.02 Mean (standard deviation, SD) calculated using age, educational level, work-related factors, and job satisfaction. In addition, the table presents the unstandardized correlation coefficients B (standard error, SE) and the standardized correlation coefficients (), which measure the type (positive or negative) and relative importance of correlation. * p < 0.05 and **p < 0.01 Source: Roelen, Koopmans, and Groothoff (2008) lan66845_06_c06_p157-190.indd 182 4/20/12 2:49 PM CHAPTER 6 Section 6.7 Quick Tips for Survey Item Construction At first glance, this looks complicated, but the more courses you have in statistics, and the more survey research you do, the easier it will be to interpret this type of data. What the researchers found with their multiple regression data analysis approach was that there are six statistically significant predictors of a person’s overall job satisfaction (based on the sample that was studied by Roelen et al., 2008). All of these predictors happen to have positive beta weights, which means the higher the value on the particular scale, the higher the overall job satisfaction. The six significant predictors (starting with the predictors with highest beta weight) are task variety, career perspectives, colleagues, working conditions, workload, and job autonomy. Note that compared to popular belief, salary is not a sig – nificant predictor of overall job satisfaction—and it is this type of insight that can make a survey design coupled with an effective data analysis strategy so powerful. As you do more work in psychology, you’ll gain experience and confidence in designing surveys as well as analyzing the results. But just how would you go about designing that survey, especially if it were the first “scientific” survey you had ever developed? We’ll discuss that in the next section. 6.7 Quick Tips for Survey Item Construction Y ou determine that closed-ended items are better suited for your research needs, and you are just about ready to start generating your item pool. But before you do that, it might be beneficial to think broadly for a moment about what you are trying to measure—that broad category of human response you are trying to capture. Consider these categories offered by eSurveyPro (2009) and Rattray and Jones (2007): (a) attitudes, beliefs, intentions, goals, aspirations; (b) knowledge or perceptions of knowledge, (c) cog – nitions; (d), emotions; (e) behaviors and practices; (f) skills or perceptions of skills, and (g) demographics. Making decisions about which broad category (or categories) you would inquire about has implications for your entire survey. For example, if you ask too many knowledge questions of your respondents, and the items are difficult, respondents may quit your survey early, not providing you with the data you need. Actual skills may be difficult to capture in a survey format, but you may be able to ask respondents about their perceptions of their own skills. Demographics can be tricky as well. Ask for too many demographics, and participants may feel a sense of intrusion. The more demographics asked, the more identifiable a participant is, even if the data are collected anonymously. Ask too few demographics and you may not be able to provide tentative answers to your hypotheses. As you have the opportunity to practice your survey skills over time, you should become more comfortable in being able to assess these broad areas. lan66845_06_c06_p157-190.indd 183 4/20/12 2:49 PM CHAPTER 6 Section 6.7 Quick Tips for Survey Item Construction Use of demographics in surveys can be problematic if not thoughtfully carried out. Sometimes, however, demographic information is vital to research. How should these surveys be handled? PR Newswire/Associated Press General advice for constructing survey items comes from many sources. The following list is a compilation of ideas from these sources: Babbie (1973), Cardinal (2002), Converse and Presser (1986), Crawford and Christensen (1995), Edwards and Thomas (1993), eSurvey – Pro (2009), Fink and Kosecoff (1985), HR-Survey (2008), Jackson (1970), McGreevy (2008), and University of Texas at Austin (2007): 1. Avoid double-barreled items. That is, each question should contain just one thought. A tipoff to this occurring is sometimes the use of the word “and” in a survey item. Example to avoid: I like cats and dogs. 2. Avoid using double negatives. Example to avoid: Should the instructor not schedule an exam the same week a paper is due? (Answered from Strongly Disagree to Strongly Agree). 3. Try to avoid using implicit negatives—that is, using words like control, restrict, forbid, ban, outlaw, restrain, or oppose. Examples to avoid: Handgun use should be banned. All abortions should be outlawed. lan66845_06_c06_p157-190.indd 184 4/20/12 2:49 PM CHAPTER 6 Section 6.7 Quick Tips for Survey Item Construction 4. Consider offering a “no opinion” or “don’t know” option. 5. To measure intensity, consider omitting the middle alternative. Example: Strongly disagree, disagree, neutral, agree, and strongly agree. 6. Make sure that each item is meaningful to the individuals being asked to com – plete the survey. That is, are the respondents competent to provide meaningful responses? Example to avoid: Xanax is the best prescription medication for clinical depression. 7. Use simple language, standard English as appropriate, and avoid unfamiliar or difficult words. Depending on the sample, aim for an eighth-grade reading level. Example to avoid: How ingenuous are you when the professor asks if you have understood the material presented during a lecture? 8. Avoid biased questions, words, and phrases. Example to avoid: Using clickers represents state-of-the-art learning technology. To what extent have clickers enhanced your learning? 9. Check to make sure your own biases are not represented in your survey items, such as through leading questions. Example to avoid: Do you think gas-guzzling SUVs are healthy for the environment? 10. Do not get more personal than you need to be to adequately address your hypotheses. Focus on “need to know” items and not “nice to know” items (helps control for survey length). 11. Try to be as concrete as possible; items should be clear and free from ambiguity. Avoid using acronyms or abbreviation that are not widely understood. Example to avoid: The DSM-IV-TR is a more accurate diagnostic tool for PSTD patients than the ICD-10. 12. Start the survey with clear instructions, and make sure the first few questions are non-threatening. Typically, present demographic questions at the end of the survey. If you ask too many demographic items, respondents may be concerned that their responses are not truly anonymous. 13. If the response scales change within a survey, include brief instructions about this so that respondents will be more likely to notice the change. 14. If your survey is long, be sure to put the most important questions first—in a long survey, respondents may become fatigued or bored by the end. lan66845_06_c06_p157-190.indd 185 4/20/12 2:49 PM CHAPTER 6 Section 6.7 Quick Tips for Survey Item Construction 15. Be sure to frame questions to minimize response set acquiescence. Ask questions that are reverse-scored (that is, strongly disagreeing is a positive outcome). Example: This course is a waste of time. (A positive answer would be strongly disagree.) Case Study: Read All About It: Sampling Matters (and Dewey Defeats Truman) American political polling has a long history dating back to 1824 (International Directory of Company Histories [IDCH], 2001), but perhaps the most famous blunder that involves the sampling of opinions from a population comes from the 1948 election where incumbent Harry S. Truman defeated the challenger Thomas Dewey. Although there had been some successes with mail-in polling in predicting presidential election outcomes in the 1930s, for the 1948 election a “perfect storm” of circumstances inter – sected to produce one of the most famous mistaken newspaper headlines of all times. At the time, George Gallup was using quota sampling, where pollsters would ask a certain number of indi – viduals from certain categories (e.g., working females, percentage of factory workers) their opinions about issues, and in particular, who they intended to vote for in the upcoming election (Jamison, 2008). After the election (and the famous blunder where the even – tual presidential election winner was declared the loser), a congressional committee chastised Gallup for not using probability sampling, which by definition would give every eligible voter in the coun – try an equal chance to be polled (IDCH, 2001). However, it was not just the misstep of selecting the wrong sampling procedure that led to this famous blunder; other events conspired to make it so. For instance, all the major pollsters (Gallup, Crossley, and Roper) stopped polling weeks before the elec – tion because major opinion changes were not expected. The Chicago Tribune, publisher of the most famous newspaper gaffe of all time, over-relied on its Washington correspondent to accurately pre – dict the outcomes. Furthermore, to get the first edition to press on time (and due to a printer’s strike at the time), the Tribune had to publish its first edition well-before election returns were known, thus preventing any last-minute changes based on early returns. Gallup also admitted after the elec – tion that he was a close friend of Thomas Dewey, and that Gallup had been in contact with Dewey throughout the campaign of 1948. All of these events coalesced into one moment where a famous national newspaper got it wrong in the front page headline on November 3, 1948 (Blackwell, n.d.; IDCH, 2001; Jamison, 2008; Walther, 2009). Reflection Questions 1. Thinking about the polling process and presidential elections today, what would be the impact of declaring victory too early for the wrong candidate? To some extent, isn’t this precisely what hap – pened in 2000 when George W. Bush ran against Al Gore for U.S. president? 2. Digging a bit deeper, would there be a way in which quota sampling could be as efficient as prob – ability sampling? What types of safeguards would need to be put into place to prevent such egre- gious errors to be drawn from survey results? 3. How does this famous incident in political history relate to the types of surveys and questionnaires that you might be asked to administer in the workplace? What lessons can be extracted from this type of sampling error that you can acknowledge and avoid if survey methodology is part of your job responsibilities someday? Associated Press lan66845_06_c06_p157-190.indd 186 4/20/12 2:49 PM CHAPTER 6 Concept Check Chapter Summary O f all the types of research you will be learning about in this course as you prepare your applied project, survey methodology may be the most valuable, because you likely will encounter surveys in the workplace, and you may be in a manage – ment position where you are asked to develop a survey or to be a savvy consumer of survey research results for your company or organization. Thus, a basic knowledge of the key aspects of survey sampling, design, scaling, and analysis could prove useful to your future. It is important to be able to distinguish between the characteristics of probability and nonprobability sampling and to know that the difference is often meaningful depend – ing on the types of conclusions you would like to draw from the data. There are a variety of approaches to survey methodology, and the design of a survey project may involve cross-sectional, longitudinal, cohort, or panel survey aspects of research design. Many scaling approaches are available, and although Likert-type scaling is prevalent, knowing the type of research question you want answered can help in the selection of the survey scale best suited for the task. There are numerous details to attend to regarding data analy – sis from surveys, and key reminders are provided in the chapter, as well as some tips for generating your own survey questions. Concept Check 1. Probability sampling means that A. the sample definitely represents the population. B. the population has multiple identifiable characteristics. C. all members of the population have an equal chance of being in the sample. D. the sample was described in sufficient detail for individual identification of members. 2. The non-random equivalent to stratified random sampling is A. cluster sampling. B. volunteer sampling. C. convenience sampling. D. quota sampling. 3. What can be both an advantage and a drawback of in-person interviews? A. The intimacy between the interviewer and participant. B. The ability of the interviewer to ask follow-up questions. C. The ability of the participant to ask clarification questions. D. The cost associated with administration. 4. In Loftus’ (1975) experiment No. 4, people were most likely to “recall” a woman pushing the baby carriage if A. the woman wore an unusual hat. B. participants were given a false presupposition. C. participants were asked a direct question. D. the baby carriage was destroyed in the video. lan66845_06_c06_p157-190.indd 187 4/20/12 2:49 PM CHAPTER 6 Key Terms to Remember 5. The most likely famous and popular scale used by psychological researchers is the A. Likert-type scale. B. visual analog scale. C. Guttman scale. D. dichotomous scale. Answers 1. C. All members of the population have an equal chance of being in the sample. The answer can be found Sec- tion 6.1. 2. D. Quota sampling. The answer can be found Section 6.1. 3. A. The intimacy between the interviewer and participant. The answer can be found in Section 6.2. 4. B. Participants were given a false presupposition. The answer can be found in Section 6.4. 5. A. Likert-type scale. The answer can be found in Section 6.5. Questions for Critical Thinking 1. Why is the survey such a prevalent methodology that is used so frequently? Does the prevalence of surveys have a negative effect on individuals answering sur – veys? Think about the number of surveys that you have received in the past two months, including telephone surveys, email surveys, mail surveys, invitations to web surveys, and so forth. How many did you answer (completely)? How might response rate temper one’s enthusiasm for the survey approach? 2. Much of the variety of survey approaches relies on stable and emerging technolo – gies. In your workplace, you may have global concerns where survey information from a specific region of the world might be valuable, but the technology infra – structure there is not as reliable as you would hope. What are your other options for gaining information about cultures and locations where technology is not so accessible? What mistakes should be avoided when looking at the application of survey methodologies as described in this chapter to other regions of the world? 3. Every methodological approach in the sciences has limitations—no approach is perfect, nor is any singular application of a methodological approach performed perfectly. What types of information are surveys good at extracting, and what types of information should be left to other types of research designs? Why? Key Terms to Remember cluster sampling The sampling practice of “clustering” groups of a population instead of evaluating each individual person to gain information when it is impossible or impractical to compile an exhaustive list of members composing the target population. cohort study A study design in which new samples of individuals are followed over time. convenience samples The sampling practice often used in exploratory research where a quick and inexpensive method is used to gather data by gathering partici- pants who are conveniently available for the purposes of data collection. coverage The issue of who has Internet access and who does not that provides a barrier to obtaining information through Internet surveys. lan66845_06_c06_p157-190.indd 188 4/20/12 2:49 PM CHAPTER 6 Key Terms to Remember coverage error An error regarding the methodology used including access to Internet, use of land lines, and other methodologies. cross-sectional survey design A study design where data collection occurs at a single point in time with the population of interest. data analysis The process of interpreting data through statistical analysis into mean – ingful and accurate conclusions. data cleaning A method of reviewing data to ensure that it has been handled and entered accurately. demographics Variables used to identify the traits of a study population. dichotomous scale A scale in which there are only two possible responses, i.e., yes/ no, male/female, true/false. Guttman scale A survey response scale that generates a set of items that increase in difficulty. If a participant agrees with one scale item, it is assumed that they agree with the preceding scale items. in-person interviews A research method – ology that allows an interviewer and a par – ticipant to build rapport through conversa- tion and eye contact, which might allow for deeper questions to be asked about the topic of interest. This presents fewer limitations about the types and length of survey items to be asked. Likert scale A survey response scale that has a 5-point scale, measuring from one pole of disagreement to the other pole of agreement with each of the scale points having a specific verbal description. longitudinal survey A study design where data collection occurs at several points over an extended period of time. measurement error An error that can occur due to a number of reasons, typi – cally including measurement variation and measurement bias. mixed-mode approach A study design where multiple research modalities are accessed to achieve the research goals. multistage sampling The two-stage sam – pling practice involving the formation of clusters as a primary selection, then sam – pling members from the selected clusters to produce a final sample. nonprobability sampling The sampling practice where the probability of each participant being selected for a study is unknown and sampling error cannot be estimated. See convenience sampling, quota sampling, snowball sampling, and volunteer sample. non-response error An error occurring when there is a response rate of 25% or less for a particular question. panel study A study design in which the same people are studied over time, span – ning at least two points in time. probability sampling The sampling prac – tice where the probability of each partici – pant being selected for a study is known and sampling error can be estimated. See simple random sampling, systematic sampling, stratified sampling, cluster sam – pling, and multistage sampling. quota sampling The sampling practice where a researcher identifies a target population of interest and then recruits individuals (non-randomly) of that popu – lation to participate in a study. representative The assumption that a sample will resemble all qualities of the general population to ensure that results of a sample can be applied to the whole general population. lan66845_06_c06_p157-190.indd 189 4/20/12 2:49 PM CHAPTER 6 Web Resources Web Resources Calculators for determining confidence intervals, sample sizes, correlations, and other research tool aids. http://www.surveysystem.com/sscalc.htm An online survey glossary that defines important research terms relevant to survey devel – opment and administration. http://k nowledge-base.supersurvey.com/glossary.htm A writing guide for survey research that assists researchers in areas such as survey development, administration, and the process of reporting results. http://writing.colostate.edu/guides/research/survey/ Resource for researchers to understand how to make proper choices in data analysis. It also contains examples of excerpts from other texts and resources. htt p://www.ats.ucla.edu/stat/stata/topics/Survey.htm Examples of best ways to construct items for survey responses so that researchers get all of the information they need for the purposes of their study. http://www.hr-survey.com/ItemConstruction.htm representativeness A challenge in Internet surveys regarding whether or not results obtained from this method is representa – tive of the entire population. sampling error An error occurring when all the potential participants from a popu – lation may not be represented in a sample. scale A tool used to measure a person’s attitudes, perceptions, behaviors, etc. that is chosen to best represent a study. semantic differential scale A survey response scale used to measure affect and emotion using dichotomous pairs of words and phrases that a participant evaluates on a scale of 1 to 7. simple random sample The sampling practice of the purest form of sampling, and probably one of the rarest techniques used where everybody in the survey population has the same probability of being tested. snowball sample The sampling practice where members of the target population of interest are asked to recruit other members of the same population to participate in the study. stratified sampling The practice of divid- ing a sample into subcategories (strata) in a way that identifies existing subgroups (such as gender) in a general population to make a sample the same proportion as displayed in a population. systematic random sample The sampling practice in which every nth person from a sample is selected. Thurstone scale A survey response scale developed to measure attitude by creat – ing a response scale of equally appearing intervals by having participants make a series of comparative judgments. visual analog scale A survey response scale used to obtain a score along a contin- uum, where a participant places a check – mark to indicate where his or her attitude or opinion falls along the scale. volunteer sample The common sampling practice where volunteers are asked to participate in a survey. volunteerism An error occurring when there is not enough self-selection in a study. lan66845_06_c06_p157-190.indd 190 4/20/12 2:49 PM 7 Key Concepts: Observation and Measurement Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • comprehend the importance of accurate and precise observation and measurement, because these operations are at the heart of what psychologists do. • explain how accurate measurements first begin with operational definitions and how these definitions must yield to measures that are both valid and reliable. • differentiate between the various types of reliability and validity. • know what information is needed to make an appropriate selection of a statistic to answer questions of interest. • recognize pitfalls to constructs graphs and avoid those pitfalls when creating their own graphs. • appreciate the various challenges and threats to collecting data for an applied project and know the steps that would lead to the initiation of such a project, such as pilot testing and data storage. • draw appropriate conclusions from empirical data and understand that psychologists seek to falsify incorrect hypotheses and never prove a theory or a hypothesis. Stockbyte/Thinkstock lan66845_07_c07_p191-228.indd 191 4/20/12 2:50 PM CHAPTER 7 Introduction Introduction T he fundamental goals of psychology are to understand, explain, predict, and con – trol behavior; thus, projects with a psychological perspective strive to fulfill one or more of these goals. Each task is a large undertaking for anyone interested in psychol – ogy and human behavior. Some – times these tasks seem daunting; it is hard to know where to start such a project. The foundations of research in psychology start with observation and measure – ment. Like building a house, if you don’t have a solid founda – tion, whatever you do afterward will be on shaky ground. So to work toward our goals in psy – chology, we acquire basic skills in observation and measurement. In previous chapters, we dis – cussed some of the major types of research designs; in this chapter, we will put the pieces together by discussing matters that are important to all research designs. Voices from the Workplace Your name: Rachel W. Your age: 20 Your gender: Female Your primary job title: University Relations Recruiter Your current employer: Whirlpool How long have you been employed in your present position? 3 months What year did you graduate with your bachelor’s degree in psychology? 2007 Describe your major job duties and responsibilities. Recruiting and attracting new talent from the university setting; interviewing; managing hiring pro- cesses; planning and implementing new strategies for gaining candidate interest and building the Whirlpool Brand presence on campuses. What elements of your undergraduate training in psychology do you use in your work? Personality analysis, counseling skills; statistics. What do you like most about your job? Meeting and working with new people. (continued) Psychological observation strives to understand, predict, explain, and control behavior. Why is it important that there are guidelines for observation and measurement? Stockbyte/Thinkstock lan66845_07_c07_p191-228.indd 192 4/20/12 2:50 PM CHAPTER 7 Section 7.1 Variables: Independent, Dependent, and More 7.1 Variables: Independent, Dependent, and More L et’s begin by reviewing the definition of a variable. A variable is an entity that can take on different values (Harmon & Morgan, 1999). In reading this text, this is prob – ably not your first encounter with the word variable. If you had any math in grade school, junior high, and high school, you should be familiar with variables. You solved equations for variable X or variable Y. You probably solved single variable equations and multivariable equations. So in math, the X variable stood for an entity that had a certain value, and your task was to figure out the value. Sometimes in an equation you might have been given X and asked to solve for Y. In very general terms, a variable is an entity that can take on different values. You’ve probably used the word variable in everyday language also, and not just in reference to math. You might have a variable speed drill press out in the shop, or you might think that the meteorologist has variable success in predicting the weather. The term variable refers to varying or different values (numbers, scores, speeds, and so on). What do you like least about your job? Sometimes the travel becomes tiring. What is the compensation package for an entry-level position in your occupation? $38,000–$60,000 What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? Typical for most business settings. What are the key skills necessary for you to succeed in your career? People skills and strategic thinking. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? Social psychology, personality, research methods, counseling. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? Internships and leadership positions within organizations. What advice would you give to someone who was thinking about entering the field you are in? Get some business background in addition to psychology. Get internship experience and get involved in organizations including the leadership roles. If you were choosing a career and occupation all over again, what (if anything) would you do differently? Nothing at this point in my career. Copyright © 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. Voices from the Workplace (continued) lan66845_07_c07_p191-228.indd 193 4/20/12 2:50 PM CHAPTER 7 Section 7.1 Variables: Independent, Dependent, and More In psychology, we build on this general definition of variable, with more specificity. We divide the variables into two broad categories—independent variables and dependent vari – ables. The key idea to remember is that a variable—either an independent variable or a dependent variable—must be able to take on different scores, numbers, outcomes, or values. Recall from Chapter 3 that the independent variable is the variable that is manipulated, con – trolled, or arranged/organized by the researcher. When the independent variable is manipu – lated or controlled, this is sometimes referred to as an active independent variable (Harmon & Morgan, 1999; Townsend, 1953). The manipulated or controlled version of the indepen – dent variable is easier to understand than the arranged/organized independent variable. The dependent variable is the one that is measured—hopefully the direct result of the manipulations of the independent variable. Dependent variables can be either qualitative or quantitative. A qualitative variable is one in which the responses differ in kind or type. That is, there is a difference in quality (what form) rather than quantity (how many), and the outcomes of these qualitative variables are usually described in words. Quantitative variables differ in amount; there is more or less of some known entity. Quantitative vari – ables are usually described by numbers, and psychologists tend to strive to develop mea – sures of behaviors (dependent variables) that yield a number. The particular approach uti – lized throughout this textbook focuses on quantitative approaches. Dependent variables can also be described in terms of the measurement process. Each quantitative approach is designed to yield a number; see Table 7.1 for types and examples of dependent variables. Table 7.1: Types of dependent variables, with examples Dependent Variable Type Examples Frequency (how often a behavior occurs) Number of cigarettes smoked in a day; number of text messages sent in an hour; number of times you studied before a test; number of times you hit the brakes as you approached an intersection Latency (the amount of time until a behavior occurs) How long it took you to learn the lyrics to a new song; after the semester started, how many days (weeks) it was until you opened this textbook; once you saw a red light, the amount of time it took until you started braking Duration (the amount of time a behavior lasts) The amount of time you spent playing XBOX 360; the amount of time you studied (in minutes); the amount of time your foot was on the brake Amplitude (the intensity of a behavior) The amount of noise (in decibels) generated by a class of third graders; the degree of test anxiety (high, medium, low) exhibited by high school students taking the SAT; the intensity of your braking (tapping the brakes versus slamming on the brakes) Choice Selection (a decision from a number of alternatives) Your answers to a multiple-choice test; your responses on a personality inventory to determine if you are introverted or extroverted; at a repair shop, which type of new brakes you select to be installed on your car When all goes well in a study, the measurements from the dependent variable are a func – tion of the independent variable; in other words, the manipulations of the independent variable lead to changes in the values of the dependent variable. These terms for variables, lan66845_07_c07_p191-228.indd 194 4/20/12 2:50 PM CHAPTER 7 Section 7.1 Variables: Independent, Dependent, and More independent and dependent, were popularized in psychology by Woodworth (1938) and later by Woodworth and Schlosberg (1954) in their first and second editions, respectively, of Experimental Psychology . The terms were used as a means of emphasizing the cause-and- effect relationship between what the researcher does (independent) and the subsequent outcome (dependent)—however, not all studies yield cause-and-effect conclusions. But where did these terms come from? And how does the use of these concepts help us to fur – ther observe and measure human behavior? It all comes down to operational definitions, which we will discuss in the next section. Classic Studies in Psychology: The Hawthorne Studies Generally speaking, the Hawthorne effect refers to the situation where participants in a study may band together to work harder than normal, perhaps because they have been specially selected for a study or they feel loyalty to the researchers or the experimental situation. The Hawthorne effect is described frequently in Research Methods texts, and Adair’s (1984) examination of texts from the 1970s and early 1980s found many erroneous descriptions of the studies. My goal is not to report similar errors here. To avoid this, in part, I consulted the original Roethlisberger and Dickson (1939) text, Management and the Worker , as well as other references on the topic. These studies began in the 1920s and ended in the 1930s. They were conducted at the Western Electric Company’s Hawthorne plant, which was adja – cent to both Chicago and Cicero. By the mid-1920s, Western Electric employed 25,000 people at the Hawthorne plant and served as the manufacturing and supply branch of American Telephone and Tele – graph, better known today as AT&T (Baritz, 1960). F .J. Roethlisberger of Harvard University and W .J. Dickson of the Western Electric Company were chiefly involved in these efforts, but many consultants were brought in over the course of the multiyear studies. In fact, there were many different studies within the “Hawthorne studies,” which dubiously leads us to the Hawthorne effect. The first set of studies, beginning in November 1924, were illumination studies conducted to examine the impact different levels of lighting had on worker productivity. In one variation, individuals were tested with lighting at 10 foot-candles (roughly speaking, 1 foot-candle is the amount of light that one candle generates 1 foot away from the candle), and over time successive work periods decreased 1 foot-candle at a time. Interestingly, when lighting was decreased from 10 foot-candles to 9 foot- candles, productivity increased. In fact, productivity continued to increase with decreased lighting until about 3 foot-candles, at which point productivity decreased (although it is reported that one employee was able to operate at the level of .06 foot-candles, or, an ordinary amount of moonlight) (Adair, 1984; Roethlisberger & Dickson, 1939). If nothing else from this study, the researchers learned from this study that understanding productivity was much more complicated than lighting. Around April 1927, a second series of studies began, which would typically be referred to as the Relay Assembly Test Room Studies (Adair, 1984; Baritz, 1960). Experimentally speaking, Roethlisberger and Dickson became more rigorous in this series of studies. For example, they selected five female employ – ees who were relay assemblers out of a large department, and placed these employees in a special test room for better control of the conditions and variables to be tested. As a dependent variable, one could measure the daily and weekly output of test relays assembled by each woman. Prior to moving the female workers into the test room, employee records were known, hence a base- line of productivity in assembling test relays was available to the researchers. Over the course of 270 weeks (yes, 270 weeks), the researchers systematically varied the conditions in the relay assembly test room, all the while recording dependent variable data on the number of test relays. These experimental variations were referred to as periods, and periods lasted weeks at a time. So, for example, for some periods the amount of voluntary rest time was increased, while for other periods voluntary rest time was decreased. For some periods the rest breaks were decreased in the morning but lengthened in the afternoon; for one period workers were giving Saturday mornings off (a 48-hour work week was customary at the time). During one period (Period XII), there was a return to baseline control conditions—a nice experimental comparison to approximately where (continued) lan66845_07_c07_p191-228.indd 195 4/20/12 2:50 PM CHAPTER 7 Section 7.1 Variables: Independent, Dependent, and More the employees started the experiment. Productivity in each period seemed to increase regardless of the manipulation introduced (Adair, 1984), in many but not all cases. In other words, when experi- mental conditions were manipulated to attempt to decrease productivity, oftentimes productivity increased. When the employees returned to baseline control conditions in Period XII, “unexpectedly, rather than dropping to preexperiment levels, productivity was maintained” (Adair, 1984, p. 336). See Figure 7.1 for the single-subject data from the Hawthorne relay assembly test room. Classic Studies in Psychology: The Hawthorne Studies (continued) Exp. Periods Relays Op. 1p Operator 1 80 70 60 40 50 12 34 56 78 9101 1121 3 Exp. Periods Weeksending Saturdays Apr 30 May28 May 25 Jun 25 Apr 27 Jun 22 Jul 23 Aug 20 Sep17 Oct 15 Nov12 Dec 10 Jan7 Feb 4 Apr28 Jun 25 Aug 18 Oct 13 Dec 8 Feb 2 Mar 3 Mar 30 May 26 Jul 21 Sep 15 Nov 10 Jan 5 Mar 2 Mar 3112 3 1927 19281929 45 67 8910 11 12 13 Op. 2a Operator 2 80 70 60 40 50 Operator 4 80 70 60 40 50 Operator 3 70 60 40 50 Operator 5 70 60 40 50 This graph shows the single-subject data from the Hawthorne relay assembly test room. Source: Roethlisberger and Dickson (1939) Figure 7.1: Single-subject data from Hawthorne study (continued) lan66845_07_c07_p191-228.indd 196 4/20/12 2:50 PM CHAPTER 7 Section 7.2 Operational Definitions and Related Ideas 7.2 Operational Definitions and Related Ideas D uring your study of psychology, you may have heard about operational defini – tions. An operational definition is a translation of the key terms of the hypothesis into measurable (i.e., public, observable) quantities. If you wanted to study depres – sion, then you would need to operationally define depression in such a way as to obtain a numerical score; or, if you wanted to measure hunger, you would have to define hunger in such a way as to get a rating or score (from a qualitative perspective, the definitions would come from non-numeric sources). Even though this makes sense, and this approach can be useful in framing how we approach measuring independent and dependent variables, it isn’t the original intention of operational definitions, or operationism. The typical focus point for the beginnings of the notion of operational definitions points to Percy Bridgman (1927), who wrote The Logic of Modern Physics . This book by Bridgman appears to have been widely read in psychology, but sometimes perhaps not far past the first few pages (Koch, 1992). To be clear, Bridgman never proposed the notion of operational definitions, but is credited with the idea of operationism (Boring, 1950). The key notion, however, was in making the connection between the behavior to be studied and the measurement of that behavior. In theory, that’s where the concept of operational definition would be so crucial to what a psychologist does. There were also additional studies as part of the Hawthorne studies, such as the Mica Splitting Test Room and the Bank Wiring Room. Taken together, what do we learn from the Hawthorne studies, and what is the Hawthorne effect? Simply put, Stagner (1982) defines the Hawthorne effect as “a tendency of human beings to be influenced by special attention from others” (p. 856). The results of the Haw – thorne studies are used by many for very different purposes, ranging from discussion of the docile worker (Baritz, 1960) to discussions about if the Hawthorne effect exists (Jones, 1992). In his review of previous studies, Adair (1984) found claims of Hawthorne effects from previous studies, and a few studies were successful when they purposely attempted to generate a Hawthorne effect. What should we take away from all this? It’s important for us to realize that when people are given special atten- tion, they may behave differently than normal. Although this effect is known as the Hawthorne effect (and it seems from the literature that Hawthorne effects have been demonstrated), it is unclear if the actual Hawthorne studies conducted in the 1920s and 1930s bear much relation at all to what we now call the Hawthorne effect. Reflection Questions 1. Think about your own prior work experience and the environment in which you worked. Did your surroundings affect your productivity? Did you work in an office where you could close the door and work in privacy, or did you work in an open space or outdoors under different work – ing conditions? How might the ready access to coworkers positively or negatively influence your productivity? 2. Consider the increasing number of individuals who telecommute and work from home. Couple this with the availability of Skype, FaceTime, and other software packages that allow for elec – tronic “face-to-face” interactions? In what types of projects or jobs might electronic interaction be sufficient? What are the conditions by which you would know you would need to meet and work with a person—in person—versus knowing that an electronic interaction would be OK? Classic Studies in Psychology: The Hawthorne Studies (continued) lan66845_07_c07_p191-228.indd 197 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process So what does this all mean? The best idea would be to remem – ber that clearly defining the key terms of psychological research is important. Measuring human behavior in a meaningful way requires a rigorous approach, striving for both reliability and validity (more on these topics in the next section). You should realize that however we choose to define the behaviors we study, there are philosophical assump – tions that underlie that method – ological approach, whether we are aware of those assumptions or not (Green, 1992). The concept of operational definitions can be a useful concept in general, but it appears to have changed from its original inception. Even so, researchers still use this notion to help develop research ideas, such as Bishop et al. (2004) working to develop an operational definition of mind – fulness. Clear, precise definitions benefit all, regardless of whether they are truly “opera – tional” or not. 7.3 The Measurement Process T he measurement process is central to any area of research. For our purposes, mea- surement involves how we capture the responses of individuals (either quantita – tively or qualitatively) in such a manner as to allow for their systematic analysis. In any measurement process, however, there is always the possibility of error. Psychologists know this, and they keep this in mind when drawing conclusions by stating them in the context of probability. In essence, whenever we measure anything, there is the potential for error. Classical test theory suggests that when a measurement is obtained, that mea – surement is composed of true score plus error (or X = t + e). Suppose you wanted to know the height of your best friend. Your best friend has a true height—in other words, there is one answer that is correct (but we don’t know the true height, so we measure). However, in measuring your best friend, there is the potential for error. You might use a yardstick or tape measure, and you could make an error in reading the number, or your friend could be wearing shoes with thick soles or slumping over. The resulting height is composed of part true score plus part error (by the way, the error could be an overestimation or an underestimation). How could we increase our confidence in minimizing the error of mea – surement? Other measurements could be taken and the results compared (a test-retest situation). Although never eliminating the potential for measurement error, error can be minimized by using the research methods of experimental psychology. The amount of error is never definitively known for a particular individual, but the amount of error is estimated when studying a group of people. If you wanted to study depression, how would you operationally define it? iStockphoto/Thinkstock lan66845_07_c07_p191-228.indd 198 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process Similarly, measuring any aspect of a person’s behavior yields a result containing true score plus error. Psychologists strive to minimize the error in measurement through the use of methodology and statistics. Where does the error in measurement error come from? A number of sources can lead to the underestimation or overestimation of the true score. A person can contribute to measurement error by being unusually motivated to do well or by not feeling his or her best. The instruments (surveys or questionnaires, for example) may be too demanding, too complicated, or too lengthy, leading to frustration, fatigue, or boredom. The researcher may also be a source of measurement error by being too friendly or too stern with the participants. The researcher may also provide inadequate instructions about the task or may simply make errors in recording participant responses. Finally, the location and specifics of the situation may lead to measurement errors; for example, the temperature, humidity, and how crowded the room is may hinder the acqui – sition of the true score. The techniques that you have learned throughout this book will help you to make better approximations of the true score while attempting to minimize the influence of measurement error. Reliability Simply put, reliability refers to consistency in measurement. If only it were that simple. If we are to have confidence that a behavior we measure is meaningful, then we have to have confidence that the measurement is reliable and consistent. There are a number of ways to think about reliability, and we’ll briefly discuss the main ones. It is important to note that reliability is estimated, not measured (Colosi, 1997). Test-Retest Reliability This type of reliability may perhaps be one of the easier types of reliability to understand. Test-retest reliability refers to the consistency in scores when the same test is administered twice to the same group of individuals. Test-retest reliability is calculated by correlating scores from each person. Test-retest reliability makes the most sense when you are trying to measure a trait or quality that is assumed to be relatively stable over time (Cohen & Swerdlik, 2005). For example, a researcher may be interested in studying the trait of humil – ity. Many personality traits are assumed to be relatively stable over time, so your humil – ity levels at the beginning of the semester should not be too different from your humility levels one month into the semester. We could then correlate your humility test score with your humility retest score; the resulting correlation coefficient is known as the coefficient of stability (Aiken & Groth-Marnat, 2006; Cohen & Swerdlik, 2005). Generally speaking, the longer the time between test and retest, the lower test-retest reliability is likely to be. Parallel Forms/Alternate Forms Reliability One of the benefits of the test-retest approach is that you create a single instrument and administer that instrument twice to the same group of people. However, one of the draw – backs to this approach is that, depending on the interval between testing, some indi – viduals might remember some of the items from test to retest. To avoid this, someone interested in constructing a reliable test could use a parallel forms or alternative forms approach. Although related, these two approaches are technically different (Cohen & Swerdlik, 2005). In a parallel forms test, you would have two versions of a test, Test A and Test B. You would then give both Test A and Test B to the same group of individuals, lan66845_07_c07_p191-228.indd 199 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process and you could correlate the outcomes between the two test administrations; this resulting correlation coefficient is known as the coefficient of equivalence (Aiken & Groth-Marnat, 2006). With true parallel forms tests, we would want identical means and standard devia – tions of test scores, but in practice, we would hope that each parallel form would correlate equivalently with other measures (Cohen & Swerdlik, 2005). With alternate forms reliability, two different forms of the test are designed to be parallel, but do not meet the criteria levels of parallel forms (for example, non-equivalent means and standard deviations). For instance, instructors often distribute two (or more) ver – sions of a test (perhaps on different colors of paper). This is usually done by the instruc – tor to minimize cheating in a large lecture hall testing situation. One hopes that the dif – ferent versions of the test (that is, alternate forms) are truly equivalent. This example provides the spirit of alternate-forms testing, but doesn’t qualify. In true alternate-forms testing, each student is asked to complete all alternate forms so that reliability estimates can be calculated. Internal Consistency Reliability Test-retest, parallel forms, and alternate forms reliability all require that a participant com – plete two (or more) versions of a measure. In some cases this may not be methodologically possible or prudent. A variety of methods have been developed to estimate the reliability of a measure in a single administration, rather than requiring multiple administrations or multiple forms. The split-half method of estimating the internal consistency of a measure involves splitting the instrument in half and then correlating the scores from the result – ing halves. For example, say I created in my Statistical Methods course a 100-item test about measures of central tendency (mean, median, and mode) and measures of vari – ability (range, variance, and standard deviation). Using the split-half method, I would ask students to take the 100-item test, but then I would separate that test into two (hopefully) equivalent halves, such as the 50 odd-numbered items and the 50 even-numbered items. I could then correlate the score from the odds and the score from the evens to obtain an estimate of internal consistency reliability (then, the correlation coefficient needs to be adjusted using a separate formula). In fact, this approach is widely used in testing (Aiken & Groth-Marnat, 2006; Cohen & Swerdlik, 2005). Interrater/Interobserver Reliability Each of the above reliability estimates focuses on participants’ responses to a test or ques – tionnaire, attempting to address, from a particular sample, the reliability of responses. Sometimes, however, an expert panel of judges is asked to observe a particular behav – ior and then score that behavior based on a predetermined rating scheme. The reliabil – ity between the scores from the raters is known as interrater reliability (also known as interobserver reliability, scorer reliability, or judge reliability; Cohen & Swerdlik, 2005). Let’s say, for example, that you are interested in the level of aggression on a playground at a local grade school. You develop a scoring system for aggressive behaviors, such as name-calling, pushing, shoving, hitting, biting, fighting, and so on. You videotape multi – ple sessions at the local playground from a location where the children cannot see the vid – eotaping (of course, you’ve gone through all the Institutional Review Board procedures to ensure that you are ethical). Then you have a panel of developmental psychologists individually view the videotapes, coding children’s behavior based on your aggressive lan66845_07_c07_p191-228.indd 200 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process behavior scale. Interrater reliability would be use – ful to determine the level of agreement between the raters in using the scoring system. There are a variety of methods you could use to calculate interrater reliability. You could first look at a percentage agreement score—that is, on all the behaviors rated, how often did your raters/ judges code a behavior in the same behavioral category? The formula you would use to calcu – late percentage agreement would be: (number of agreements/number of agreements and dis – agreements) × 100. Another technique, with two judges, is that you could simply calculate a cor – relation coefficient between the pairs of scores on each behavioral instance. There are multiple approaches to capturing the consistency of raters and judges in these situations. Validity Whereas reliability addresses consistency in mea – surement, validity addresses the question “Are we measuring what we think we are measur – ing?” There are at least two major approaches to how we think about validity. One approach comes from the psychometric literature and how psychologists construct new measurement instruments. The classic approach here is to discuss content validity, construct validity, criterion-related validity, and face valid – ity. The other approach comes from our study of experimental design and particular quasi-experimental designs—in fact, some refer to this latter approach as a “Cook and Campbell” approach, in part due to an influential book (1979) that brought together this conceptualization of validity, as well as a classic listing of threats to validity. We’ll briefly review both major approaches here. In the classic psychometric approach, there is a trio of C’s: content validity, criterion- related validity, and construct validity. Note that for a measure (an instrument, survey, questionnaire, test) to have validity, all three types of validity mentioned here are impor – tant; each is necessary, but not sufficient alone, to establish validity (Morgan, Gliner, & Harmon, 2001). Content validity refers to the composition of items that make up the test or measure. Do the contents of the test adequately reflect the universe of ideas, behaviors, attitudes, etc., that compose the behavior of interest? For example, if you are interested in studying introversion, and you are developing an introversion inventory to measure one’s level of introversion, do the items on the inventory capture the totality of the concept of introversion? More formally, “content validity is concerned with whether the content of a test elicits a range of responses that are representative of the entire domain or universe of skills, understandings, and other behaviors that a test is designed to measure” (Aiken & Groth-Marnat, 2006, p. 97). If you are taking the Graduate Record Exam (GRE) subject Interrater reliability can be used to determine the level of agreement between raters who used a scoring system to find out the level of aggression on a playground. Science Faction/SuperStock lan66845_07_c07_p191-228.indd 201 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process test in psychology, content validity asks the question “Are the items truly capturing your knowledge of psychology?” Content validity alone does not mean that an instrument is valid (remember, each type of validity is necessary but not sufficient). Instead, content validity is established through the process of creating test items, including a thorough review of the literature and consultation with experts (Morgan et al., 2001). Criterion-related validity refers to how the measurement outcome, or score, relates to other types of scores. A general way to think about criterion-related validity would be “Given that we now have a score that is reliable, what will this score predict?” Psycholo – gists are very interested in making predictions about behavior, so criterion-related validity can be very useful in practice. Two subcategories of criterion-related validity are concur – rent validity and predictive validity. Concurrent validity refers to how the score on a test or inventory is related to your current state of affairs. For example, if a person were to take a mental status exam right now, and this person scores in a certain range, this might tell us right now (concurrently ) that this person is suffering from a particular type of mental dis – order. Or, when you go to take your driver ’s license test, and you receive a passing score, this indicates (hopefully) that you possess current knowledge of safe driving practices. Predictive validity takes current knowledge and attempts to make a prediction about the future, such as a college admissions office using high school GPA as one of the predictors of future success in college, or the scores on a pre-employment test attempting to predict whether you will be a good hire and if you will become an effective manager. Essentially, criterion-related validity addresses the predictability of current events or future events. Construct validity has been called “umbrella validity” (Cohen & Swerdlik, 2005, p. 157) because all types of validity feed into the overall conclusion about construct validity. Gen – erally speaking, construct validity exists when a tests measures what it purports to mea – sure. A construct is a hypothetical idea that is intended to be measured but does not exist as a tangible thing. For example, intelligence is a construct. That is, intelligence is this hypothetical idea that we believe humans and animals possess in certain degrees. If we were to do a post-mortem examination of a person’s brain, we would not be able to extract the part of the brain known as intelli – gence—intelligence is not a tangible, physical entity. Intelligence is a hypothetical idea that psychologists (and others) construct, and we spend considerable time and energy measur – ing this hypothetical idea. Much of what we study in psychology are constructs such as humility, sympathy, depression, happiness, anxiety, altruism, success, dependence, and self-esteem. But to accumulate evidence in support of approaching construct validity, Cohen and Swerdlik (2005) suggest the fol – lowing steps: (a) Establish that the test mea – sure one singular construct; (b) test scores change as a function of how a theory would predict they change, such as increasing With construct validity, intelligence is a hypothetical idea that psychologists spend considerable time and energy measuring. Stockbyte/Thinkstock lan66845_07_c07_p191-228.indd 202 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process or decreasing with age, time, or an experimental manipulation (independent variable); (c) using a pretest-posttest design, test scores change in a predictable and theoretically relevant manner; (d) test scores among different groups of people differ in a theoretically relevant way; and (e) test scores correlate with scores from other tests; these scores are pre – dicted theoretically to be related, and they are related in the manner predicted by theory. Although not part of the “C” trio of validities, face validity is often mentioned as a type of validity, referring to whether the person taking the test believes that it measures what it purports to measure. Face validity is ascertained from the perspective of the test taker, and not from the responses to test items. If the test-taker believes that the items are unrelated to the stated purpose of the test, this might affect the quality of responses, or our confi – dence that the test was taken seriously. Face validity may be more relevant to the “public relations” of the test, rather than the test results themselves (Cohen & Swerdlik, 2005). The 3 C’s of validity (plus face validity) compose the classic test construction approach to validity. These are important concepts to consider when developing a measure of behav – ior. But there are other considerations as well, such as the level of confidence we have in the conclusions we draw or the generalizability of the results from the present study to other times, places, or settings. Cook and Campbell (1979) offered a different conceptual – ization of validity, and these ideas are particularly relevant. Four different categories of validity include internal, external, statistical conclusion, and construct validity (Cook & Campbell, 1979). In fact, these authors conceptualize validity a bit differently from psychometricians when they define validity as “the best available approximation to the truth or falsity of propositions, including propositions about cause” (p. 37). Internal validity refers to the general nature of the relationship between the inde- pendent variables and the dependent variables. The chief goal in establishing internal validity is the determination of causality—did the manipulation of the independent vari – ables cause changes in the dependent variables? External validity refers to the question of whether, if a causal relationship does exist between these variables, the relationship can be generalized to other research settings, other samples, or other times. Statistical conclu- sion validity refers to the sensitivity or statistical power of the experimental situation. In our attempt to determine cause-and-effect relationships, are we using both method – ological and statistical approaches sensitive enough to capture causal relationships, if they exist? Finally, construct validity concerns how the operations used in measurement are related to the higher-order constructs upon which they presume to measure. For example, researchers may develop a test to measure intelligence, and this new test may have inter – nal, external, and statistical conclusion validity, but does the test truly measure intelli – gence? This is the question construct validity attempts to answer; in this case, construct validity overlaps with the psychometric approach to validity in asking the question “Are we measuring what we think we are measuring?” One of the benefits of the Cook and Campbell approach to validity is that it provides not only a framework to evaluate research but also insight in the adequate design of research before it is conducted. Cook and Campbell (1979) painstakingly listed many of the threats to each of the four types of validity. We won’t re-create that entire listing here, but by thinking about the threats to internal validity, for example, you can begin to see many of the factors that are relevant to designing a study as well as conducting the study. Our goal, of course, is to design a research study in such a way as to avoid or minimize threats to validity. Table 7.2 lists the classic threats to validity, with a brief definition and example. lan66845_07_c07_p191-228.indd 203 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process Table 7.2: Classic threats to internal validity Threat to Internal Validity Brief Definition Research Example HistorySomething happens during the experimental session that might change responses on the dependent variable. If you are collecting data in a large classroom and the fire alarm goes off, this may impact participants’ responses. Maturation Change in behavior can occur on its own due to the passage of time (aging), experience, etc., with the change occurring separate from the independent variable manipulation. In a within-subjects design, you have participants view visual information on a computer screen in 200 trials. By the end of the study, participants may be fatigued and changing dependent variable responses due to time and experience. Testing When testing participants more than once, earlier testing may influence the outcomes of later testing. If you design a course to help students do better on the GRE, students take that test at the beginning of the course and again at the end of the course. Mere exposure to the GRE the first time may influence scores the second time, regardless of the intervention. Instrumentation A change occurs in the method by which you are collecting data; that is, your instrumentation changes. If you are collecting data through a survey program on a website, and the website crashes during your experiment, then you have experienced an instrumentation failure. Statistical Regression When experimental and control group assignments are based on extreme scores in a distribution, these individuals at the extremes tend to have scores that move toward the middle of the distribution (extreme scores, when they change, become less extreme). In a grade school setting, children who are scoring the absolute lowest on a reading ability test are given extra instruction each week on reading and are retested after the program is complete. Because these children’s scores were at the lowest part of the distribution, these scores, when they change, have nowhere else to go but up. Is that change due to the effectiveness of the reading program, or statistical regression? Selection When people are selected to serve in different groups, such as an experimental group and a control group, are there pre- existing group differences even before the introduction of the independent variable? In some studies, volunteers are recruited because of the type of study or potential implications (such as developing a new drug in a clinical trial). Volunteers, however, are often motivated differently than non-volunteers. This pre-existing difference at the point of group selection may influence the effectiveness of the independent variable manipulation. Mortality Individuals drop out of a study at a differential rate in one group compared to another group. In your study, you start with 50 individuals in the treatment group and 50 individuals in the control group. When the study is complete, you have 48 individuals in the control group but only 32 individuals in the experimental group. There was more mortality (“loss of participants”) in one group than the other, which means there is a potential threat to the conclusions we draw from the study. (continued) lan66845_07_c07_p191-228.indd 204 4/20/12 2:50 PM CHAPTER 7 Section 7.3 The Measurement Process Table 7.2: Classic threats to internal validity (Continued) Threat to Internal Validity Brief Definition Research Example Interaction with SelectionIf some of the above threats happen in one group but not in the other group (selection), then these threats are said to interact with selection. If the instrumentation fails in the control group but not in the experimental group, this is known as a selection × instrumentation threat. If something happens during the course of the study to one group but not the other, this is a selection × history threat. Diffusion/ Imitation of Treatments If information or opportunities for the experimental group spill over into the control group, the control group can obtain some benefit of an independent variable manipulation. In an exercise study at the campus recreation center, students in the experimental group are given specific exercises to perform in a specific sequence to maximize health benefits. Control group members also working out at the Rec Center catch on to this sequence, and start using it on their own. Compensatory Equalization of Treatments When participants discover their control group status and believe that the experimental group is receiving something valuable, control group members may work harder to overcome their underdog status. In a study of academic achievement, participants in the experimental group are given special materials that help them learn the subject matter better and perform better on tests. Control group members, hearing of the advantage they did not receive, vow to work twice as hard to keep up with the experimental group and show them that they can do work at equivalent levels. Resentful Demoralization When participants discover their control group status and realize that the experimental group is receiving something valuable, they decide to give up and stop trying as hard as they normally would. In the same academic achievement example as above, rather that vowing to overcome their underdog status, the control group simply gives up on learning the material, possibly believing that the experiment is unfair and wondering why they should bother to try. The ideas of reliability and validity are central to both the observation and measurement of behavior. These are everyday ideas that psychologists utilize to improve their work. To conclude our discussion of those ideas here, let me leave you with a practical example of how researchers use both ideas of reliability and validity in their measure of behavior. Let’s say you are working in a drug rehabilitation clinic where you are helping people overcome an addiction to cocaine. Knowing when someone’s craving for cocaine is ele – vated in this setting could be important; you might want to be more vigilant with the person, provide greater assistance, offer more intense counseling, and so on. There are multiple scales that measure cocaine craving, but they are rather long. There is a 45-item Cocaine Craving Questionnaire-Now scale and a 33-item Questionnaire of Cocaine Use scale (see Sussner, Smelson, Rodrigues, Kline, Losonczy, and Ziedonis (2006) for more details about these scales). But administering such long surveys in the midst of someone’s recovering from a cocaine addiction might be unwieldy. So Sussner et al. (2006) set out to create a shorter cocaine craving questionnaire that would be easier to administer but at the same time possess both validity and reliability. lan66845_07_c07_p191-228.indd 205 4/20/12 2:50 PM CHAPTER 7 Section 7.4 Scales of Measurement and Statistic Selection Sussner et al. (2006) called their new instrument the Cocaine Craving Questionnaire— Brief. It is a 10-item survey and was derived from the longer CCQ-Now 45-item survey. The key question becomes this: Does the new scale possess reliability and validity? To establish validity, these researchers correlated scores on the CCQ-Brief with pre-existing valid measures of cocaine cravings, and high positive correlations between previous measures and new measures were taken as one indicator of validity. That is, if you create a new measure, and your new scores correlate highly with a measure that has already been shown to be valid, then your new measure begins to accumulate evidence of validity. As for reliability, the CCQ-Brief was studied using an internal consistency (inter-item approach), yielding a Cronbach’s α = .90. Thus validity and reliability are regular components of the research process, as demonstrated by the Sussner et al. (2006) study. One other note to make about the relationship between validity and reliability— an instrument can be reliable without being valid, but an instrument can only be valid when it is measured reliably. We understand the importance of measuring behavior reliably and in a valid manner, but how do we collect data? When we rely on numerical (quantitative) scores, what do the actual numbers mean, and how might we analyze the data once we have it? 7.4 Scales of Measurement and Statistic Selection T his chapter is about observation and measurement. We already know that when we measure the dependent variable in a quantitative fashion, we want numerical scores that are both reliable and valid. But how do we obtain those scores—that is, how do we measure human behavior? The process of translating observations into scores involves scales of measurement . Based in part on a seminal article by Stevens (1946), there are four general scales of measurement: nominal, ordinal, interval, and ratio. This order of presentation is important, because it is generally thought that the nominal scale has the least utility in terms of value and statistical analysis options, and the ratio scale has the most utility and greater statistical options. Said another way, we’d prefer to have ratio scale data than nominal scale data in most situations. But before we address data analysis options, let’s review a bit about each type of scale of measurement. Nominal Scales On the nominal scale, individuals are placed (or coded) into classifications or categories that are used to keep track of similarities and differences. For example, each basketball player on the court wears a different number on his or her jersey to help others keep track of the players. A higher jersey number does not mean that the player is better, nor does a lower jersey number mean that the player is worse. The numbers themselves do not express relative value, but the numbers are used to track differences. Numbers can also be used to track similarities. For example, if we were interested in conducting a poll on campus about who you plan to vote for in the next presidential election, we might also want to ask prospective voters about their political affiliation (Republican, Democrat, or Independent). As we recorded this information, we might code the data in such a way as 1 = Republicans , 2 = Democrats , and 3 = Independents . Note that here the use of numbers is to classify people into similar categories, and different numbers are used to denote lan66845_07_c07_p191-228.indd 206 4/20/12 2:50 PM CHAPTER 7 Section 7.4 Scales of Measurement and Statistic Selection different political affiliations. The numbers them – selves do not have implicit meanings; that is, Independents are not one and one-half times bet – ter than Democrats, nor do Republicans have half the value of Democrats. The numbers themselves are arbitrary placeholders allowing us to keep track of differences; we could have just as easily coded 14 = Republicans , 3 = Democrats, and 77 = Independents ; the numbers selected are arbitrary, used to classify those in a similar category. Why would we want to classify nominal scale cat – egories with numeric labels? This process facili – tates data analysis in statistical programs such as SPSS (Statistical Program for the Social Sci – ences). But only certain types of analyses are rel – evant with nominal scale data. Take, for example, the last four digits of your cell phone number. These data are nominal scale data. The last four digits are used to keep track of different phone accounts, but a higher phone number like x8783 does not mean you have a better number than x2334. Those four digits just help keep track of different telephone accounts and lines. However, we could ask a classroom of students to provide us with the last four digits of their cell phone number and then calculate the average phone number (try this sometime with your classmates). You can do this on a calculator, and SPSS will do it for you as well. However, calculating the mean of nominal scale data doesn’t make a lot of sense—with our two numbers above, the average phone number is 5558.50, which doesn’t mean much. You will need to know the appropriate data analysis techniques for different scales of measurement; later in this chapter there is some guidance on these issues. If you truly wanted to have an idea about the central tendency of nominal scale data, the mode would be a better choice. The mode is the most frequently occurring score in the distribu – tion. With our political affiliation example, you might discover that code 2 (Democrats) is the most frequently observed political affiliation on campus. It doesn’t make any sense to average together the 1s, 2s, and 3s, but it is meaningful to know that 2 is the modal score. Ordinal Scales On the ordinal scale, the magnitude of the numbers mean something—in other words, a higher number means more, and a lower number means less. There is an underlying con – tinuum expressed with the numbers on the ordinal scale. One example would be when items are rank ordered. If the data are rank ordered in some way, then you are dealing with ordinal scale numbers. Another assumption of the ordinal scale is that the distance or difference between adjacent numbers is not assumed to be equal; in fact, we assume unequal intervals. Similar to how jersey numbers help fans keep track of their favorite players, nominal scales help researchers categorize study participants. Comstock/Thinkstock lan66845_07_c07_p191-228.indd 207 4/20/12 2:50 PM CHAPTER 7 Section 7.4 Scales of Measurement and Statistic Selection On the ordinal scale, numbers are unlike the arbitrary values on the nominal scale. So, if you were to rank order your top 10 movies of all time, the No. 1 movie would be your most favorite, and your No. 10 movie would be your tenth favorite. In this rank order scenario, the lower the number, the better the movie—the number has meaning. Anything with rank order is ordinal scale: your class rank when you graduated from high school, the national rankings of college football BCS polls, the gold-silver-bronze medals of the Olympic Games, and so on. We can analyze ordinal scale statistics, and there are many techniques available. The statistical approaches utilized to analyze both nominal and ordi – nal scale values fall under the heading of nonparametric statistics. This term refers to the idea that the data from nominal and ordinal scale measurements may not necessarily be normally distributed, hence specialized statistical procedures are used (more later on how the underlying assumptions of the data influence the statistical approach). One last thought about the ordinal scale: The intervals are assumed to be unequal. Do you remem – ber when Michael Phelps from the United States won his Olympic Gold medal in 2008 by .01 of a second? In the men’s 100m fly, Michael Phelps won the gold with a time of 50.58 seconds, Milorad Cavic of Serbia won the silver with a time of 50.59 seconds, and Andrew Lauterstein won the bronze with a time of 51.12 seconds. The distance between the first place and second place medals was .01 second, while the distance between the second and third place medals was .13 seconds. This is what is meant by uneven intervals—the distance between first and second place is not necessarily the same as between second place and third place. Interval Scales The interval scale builds on the properties of the ordinal scale (Stevens, 1946). So, on the interval scale, the numbers are meaningful. Higher numbers mean something; that is, there is a continuum underlying the number system. In our typical thinking about the interval scale, the number zero is just another number on the scale. Another new addition to our thinking about the interval scale is that the intervals are now uniform and meaningful (thus, the interval scale). One good example of the interval scale (although not overly psycholog – ical in nature) is the Fahrenheit scale. A higher number means more heat, and a lower number means less heat. The intervals are uniform and meaningful— the distance between 20 8 and 408 is the same distance between 508 and 70 8. Finally, 08 does not mean lack of heat on the Fahren – heit scale; it’s just another num – ber on the scale. There are other examples of interval scales, but they don’t tend to be psycho – logical in nature—latitude and The Fahrenheit scale on a thermometer is a good example of an interval scale. iStockphoto/Thinkstock lan66845_07_c07_p191-228.indd 208 4/20/12 2:50 PM CHAPTER 7 Section 7.4 Scales of Measurement and Statistic Selection longitude, altitude, a person’s net financial worth, clothing sizes, etc. But what about psy – chological variables? The challenge for the interval scale with psychological variables deals with zero as just another number on the scale. For the next and final scale of measurement (ratio scale), zero is the absence of value, which makes ratios meaningful. But on the interval scale, zero is typically thought of as just another number on the scale. Let’s say you know some – one who scored a zero on an intelligence test. Is zero a legitimate score on an intelligence test? Are negative values possible? More importantly, would a zero on an intelligence test imply a lack of intelligence? It’s hard to know what a zero means on a test of intelligence. In actual practice, there may not be many true psychological interval scales. Most psy – chologists make the assumption, then, that these types of numeric scales with equal inter – vals are treated the same as ratio scales. In fact, in some places you’ll hear these types of data referred to as interval/ratio scale data. In SPSS, for example, the only options when you select a scale for your variable data are nominal, ordinal, and scale. Of course, there are some dangers to this type of assumption (Labovitz, 1967). Interval and ratio scale data also fall under the category of parametric data, which means additional assumptions are made about the underlying distribution of the data gathered—that is, that the distribution is relatively normal (if the data are not normally distributed, it will have implications for the conclusions drawn from the analysis, as well as might dictate that a non-parametric approach be used). But in actual practice, even though there are concerns and limitations, many psychological variables are considered interval/ratio (or scale) variables. Ratio Scales On the ratio scale, many of the characteristics previously mentioned still apply. In fact, the ratio scale is our usual use of numbers. There is a quantitative dimension and an underly – ing continuum for the numbers used, and on the ratio scale, zero is used to identify the lack of something (not just another number on the scale like in true interval scales). When zero is the endpoint on the scale, ratios are now meaningful. For example, 10 inches is twice as long as 5 inches (a ratio), because 0 inches means no length. Four hours is half as much as 8 hours because 0 hours means no time. When 0 means the lack of value, then ratios become meaningful. Ratios are not meaningful, however, on the interval scale. Is 208 twice as warm as 10 8? On a psychological test of intelligence, is someone who has an IQ of 120 twice as intelligent as someone who has an IQ of 60? When 0 means something (that is, when 08 F is just another number on the scale) then ratios are difficult to interpret meaningfully. So think of ratio scale data as our usual use of numbers, such as counting the frequency of a behavior or asking a person to respond on a scale from 1 to 10. We certainly make assumptions about what these numbers mean, and because the interpretation of true interval scale data is difficult for psychological variables, often in practice we lump together interval and ratio, sometimes referring to this type of data as interval/ratio. In fact, some would say that true ratio scale data are rare in psychology (Becker, 1999). Means and standard deviations make sense with interval/ratio scale data, but not with nominal or ordinal scale data. Oftentimes, psychologists take advantage of these fuzzy boundaries between scale types. As discussed in chapter 6, a very common scale used in survey research is a Likert-type agreement scale, where the items are declarative lan66845_07_c07_p191-228.indd 209 4/20/12 2:50 PM CHAPTER 7 Section 7.4 Scales of Measurement and Statistic Selection statements and you are asked to respond on a scale such as 1 = strongly disagree , 2 = dis- agree , 3 = neutral , 4 = agree , and 5 = strongly agree . We treat these responses as interval/ ratio scale data when we calculate the mean response for any particular item. However, when carefully examined, these data are not ratio scale and probably not interval scale, but rather ordinal scale. Let’s say that two instructors are being evaluated at the end of the semester, and one of the items on the course evaluation is “This instructor seemed well prepared for class.” Dr. A might receive a mean score of 4.22, whereas Dr. B receives a mean score of 3.75. We treat these data like interval/ratio data, but the reported score of a value between 1 and 5 is more similar to a rank order score. For example, is the dis – tance on this scale between 2 and 3 the same as the distance between 4 and 5 (remember, on the ordinal scale, intervals do not need to be uniform; but on the interval and ratio scales, they must be). So why would studies treat ordinal scale “agreement data” like interval/ratio data? One answer is that the types of statistics that are applied to interval/ratio scale data are taught more, and are more familiar to most psychologists, such as the t test or ANOVA, as opposed to non-parametric statistics used with ordinal scale data. Also, it’s easier to understand and interpret the means of interval/ratio scale data, rather than the median rank orders of ordinal scale data. For now, you just need to know that we make multiple assumptions about data analysis, and sometimes we violate those assumptions. But how would you determine what statistic to use in which data analysis situation? To answer that question, we need to know (1) what types of scales your variables are measured on, and (2) what type of conclusion you want to draw. There are a number of ways to approach this complex issue (Morgan, Gliner, & Harmon, 2002; Vowler, 2007); a broad approach would be to ask if you are interested in examining the differences between groups or the associations or relationships among variables. If you are interested in understanding the differences between groups, you may take an approach where you would use a t test or F test from an ANOVA. If you are more inter – ested in associations, you might be using a chi-square, correlation, or multiple regres – sion approach. However, we need more answers to more questions prior to proceed – ing. It would be good to be able to clearly identify our independent variables and our dependent variables—not as easy as it might seem, especially for beginning students. We would also need to know about the type of research design utilized—for example, between groups design, within groups design, and mixed group design,. When examin – ing our variables (both independent and dependent), we’d want to know their scales of measurement (nominal, ordinal, interval/ratio)—sometimes the distinction you might hear is continuous variable versus discrete variable. There are also assumptions that underlie not only the data collection process but also the requirements of certain data analysis approaches (a common assumption is that the data being analyzed are nor – mally distributed). So there are many considerations when selecting the appropriate statistic, and what we’ve discussed here is just a brief overview. If you continue in psychology, and especially if you continue doing research, your confidence will likely grow in knowing which statistic to use in which situation. lan66845_07_c07_p191-228.indd 210 4/20/12 2:50 PM CHAPTER 7 Section 7.5 Graphing Your Results 7.5 Graphing Your Results A fter you’ve completed your observation and measurement of behavior, eventually it will be time to tell your story. Storytellers have many conventions that they can use to best communicate a story, such as foreshadowing, building action, conflict resolu – tion, etc. In thinking about scientific storytelling (for more, see Landrum, 2008), a graph can help tell a complicated story in an efficient manner. Tufte (1983) suggests that clear, precise, and efficient graphs should: (a) show the data; (b) encourage the viewer to think about the content of the graph rather than focus on the graphic design; (c) avoid distortions; (d) present much data in a small space, making large data sets more coherent; (e) reveal the complexity of the data on both a broad level and fine level; (f) serve a clear purpose—description, explora – tion, tabulation, or decoration; and (g) have close integration with the text that accompanies the graph. Not that undergraduates have a lot of spare time, but if you are truly interested in a classic about graphical design, read Tufte’s (1983) The Visual Display of Quantitative Information —it is a true classic. But as you can see from the previous paragraph, designing a graph that con – forms to all of these characteris – tics is a fairly tall order, which is why graphs may not be used as much in psychology journals. However, the lack of graphs in psychology writing could have a detrimental effect. When Smith, Best, Stubbs, Archibald, and Roberson-Nay (2002) looked at perceptions of hard and soft sciences, these researchers found that a higher usage of graphs contributes to the perception of a hard science, whereas the greater usage of tables and inferential statistics contributes to the perception of a soft science. One suggestion would be to not select a graph depending on its perception of hard or soft, but to think about this: Does the graph help explain a complicated story with clarity, precision, and efficiency? (Tufte, 1983). In creating your graph, you must also be fair with the data. There are many good guides that can help you with this, including your Publication Manual (APA, 2010), as well as Nicol and Pexman’s (2010) Displaying Your Findings: A Practical Guide for Creating Figures, Posters, and Presentations . Kosslyn (1994) also offered sage advice, with excellent examples; for instance, he presented the graph shown in Figure 7.2 to depict the number of warheads in possession of the United States and (then) U.S.S.R. in 1991. The actual data at the time were that the United States had 11,877 warheads and the U.S.S.R. had 11,602 warheads. You should present your data in an efficient manner, using clear, concise charts, graphs, and tables. iStockphoto/Thinkstock lan66845_07_c07_p191-228.indd 211 4/20/12 2:50 PM CHAPTER 7 Section 7.5 Graphing Your Results United States U.S.S.R. 12 10 8 6 4 2 0 Number of strategic warheads (thousands) Here is an example of a graph where the data are accurate, but because of the scale selected for the Y (vertical) axis, it makes it appear that there is not much difference in the number of warheads in the two countries. Source: Statistical Program for the Social Sciences Figure 7.2: Number of warheads, version 1 Depicting the data in this purposeful way tells a particular story, and in this case, the author would be emphasizing the near equivalence of the number of strategic warheads in possession by both entities. But look at the same data as graphed in Figure 7.3. This presentation certainly tells a different story. United States U.S.S.R. 12.0 11.9 11.8 11.7 11.6 11.5 Number of strategic warheads (thousands) Here the data are accurate, but because of the scale selected for the Y (vertical) axis, it appears that there is a large difference in the number of warheads in the two countries. Source: Statistical Program for the Social Sciences Figure 7.3: Number of warheads, version 2 lan66845_07_c07_p191-228.indd 212 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions This graph clearly emphasizes differences. But truncating the x-axis (the zigzag line on the axis), this bar graph is meant to send a message. Of course, you’ll want to use a graph to send a message, but you want to be fair with the data. So if you truncate an axis, be sure to label it properly. Try not to distort the graph to tell the story—ideally, the data are telling a compelling story, and the graph is the mechanism you choose for effective communica – tion. For tips on graph creation, see the following list. 1. Use bright white paper and black ink in drawing graphs. 2. The ordinate (y axis, vertical line) always depicts the dependent variable, and this line should be about 2/3 the length of the abscissa (x axis, horizontal line) which always depicts the independent variable. For every inch up the ordinate, the abscissa should be 1.5 to 1.6 inches long. 3. Label both axes and provide a figure legend in the graph if necessary; the figure caption is placed on a separate page. 4. In its final form, the lettering on the graph should be no smaller than 1/16 of an inch. 5. The demands of your audience (e.g., your instructor or a professional journal) may dictate other procedures for creating acceptable graphs. When in doubt, con – sult the Publication Manual (2010) or Nicol and Pexman (2010). Even when accurate, the depiction of data can be manipulated in a number of ways. It is important to attend to the details of graphs so that you can draw your own conclusions about their meaning. 7.6 Procedural Matters: Experimental Pitfalls and Precautions A s you go about designing a study and collecting data for analysis and in terpreta – tion, you will want to think ahead about some of the general issues involved in drawing conclusions from data. There are some that apply to all of the specific designs we discussed earlier in this book. Confounds In designing and conducting your study, you will want to avoid confounds or confound – ing variables as much as possible. As a reminder, a confound is a complication in the research design, based on the idea that something outside of the plan of the experiment has influenced your measurement of the dependent variable in addition to (or in place of) your independent variable. In other words, a confound means something else may account for your results, other than the purposeful design of your study. For example, a confound might influence one level of your independent variable manipulation, but not the other levels. Let’s say that you were interested in testing the effectiveness of a new teaching technique in the classroom. For comparison purposes, you want to teach the same course to the same level of students, grade the same way—in fact, try to do everything the same as much as you possibly can except to manipulate the independent variable, the teaching technique (perhaps traditional lecture style versus a service learning approach). You then decide that lan66845_07_c07_p191-228.indd 213 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions the best way to do this is to find a course big enough that offers two similar sections taught by the same instructor, but the sections are taught at different times of the day. After your study you find that the service learning group scored significantly better on tests, so you conclude that it is the better technique, right? Well, there may be a confound. It may be that time of day confounds with your dependent variable. What if one section is taught at 8 a.m. and the other section at 2 p.m.? Is it reasonable to assume that performance might already be different between these two groups before the introduction of the independent variable? If you answer yes, then time of day is a potentially confounding variable. How do we handle confounds? If you can show that previous studies ruled out these con – founding variables influencing your dependent variable, then you can be more confident in your results. You could try to find two sections closer in time to minimize any confound – ing. You could expand the study and incorporate time of day as an independent variable, with the aid of other instructors teaching the same types of multiple-section courses. Con – founds are not necessarily fatal flaws, but they do detract from drawing strong conclu – sions from your study. While a confound may only influence one level of the independent variable, an artifact influences all levels of the independent variable. Confounds threaten internal validity, whereas artifacts threaten external validity. Artifacts When a data collection artifact occurs, the measurement process is distorted, biased, or corrupted in some fashion. In fact, it is not often known in what direction the distortion may be (it may inadvertently support the experimenter ’s hypothesis or detract from it)— in essence, we do not know if the artifact is leading us to a Type I or Type II error. The four general categories of artifacts to be presented here include physical setting, within subjects, demand characteristics, and experimenter expectancy. Physical Setting In some cases the physical set – ting may influence participant performance and lead to data artifacts. Too warm, too cool, too humid a setting may detract from participant’s true perfor – mance. Noise, general atmo – sphere, and crowdedness may also influence performance. By being sensitive to these condi – tions experimenters can usually provide an adequate atmosphere for participants. If some of these conditions are out of the experi – menter ’s control, then consis – tency is the goal: If you believe that the room temperature may affect participant performance, then test all participants at the The physical setting of your study must be carefully chosen because it may cause data artifacts. What might be problematic about this setting? iStockphoto/Thinkstock lan66845_07_c07_p191-228.indd 214 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions same temperature, or turn the potential data collection artifact into an independent vari – able and systematically test the hypothesis of whether or not the physical setting variable effects participants’ performance. If you think that room temperature is affecting perfor – mance on a task, then you could test the hypothesis empirically. Try to arrange for three different rooms, each at a different temperature, and then test to see if temperature does indeed influence task performance. Within Subjects There are a number of subject-related artifacts to be aware of when collecting data. (In prior editions of the APA Publication Manual , the human participants in a study were called subjects. Even though they are called participants today, the term “subject” is still used in some cases, such as a within-subjects design.) In particular, response sets can influence participant performance. A response set is a pattern of responding seen in a participant that may not accurately reflect the participant’s true feelings on a topic. For example, response set acquiescence involves the participants getting stuck in saying yes repeatedly in a survey or questionnaire. If participants see their own pattern of respond – ing as all yeses, then they may stop reading the questions carefully and answer yes to everything (of course, the way to avoid this is to have questions worded in both direc – tions; that is, to have both yes and no answers indicate whatever measure of interest you are studying in your experiment). See Table 7.4 for an example of how to avoid response set acquiescence. Table 7.3: Sample survey items’ susceptibility to response set acquiescence Susceptible to Response Set Acquiescence Less Susceptible to Response Set Acquiescence 1. The instructor held my attention during class lectures. 2. The instructor wrote exams that fairly tested the material covered. 3. The instructor seems to be well prepared for class. 4. The instructor was available for extra help outside of class. 5. The instructor regularly answered students’ questions. 1. The instructor was seldom able to hold my attention during class lectures. 2. The instructor wrote exams that fairly tested the material covered. 3. The instructor often appeared to be unprepared for class. 4. The instructor was available for extra help outside of class. 5. The instructor rarely answered students’ questions. Note. These items could be answered on a scale from 1 = strongly disagree to 5 = strongly agree. Response set social desirability comes from participants’ responding in a pattern that they believe makes them look good, or look better than they are. That is, participants are presenting themselves as socially desirable when, in fact, they may not be. If you were to ask participants if they are racist, you would probably obtain an underestimation of the actual number of people who could be considered racist. With socially charged issues it is often difficult to overcome response set social desirability, but with carefully worded questions and multiple approaches to the concept (such as role-playing or simulations), such issues can be studied effectively. Also, there are scales that are used to attempt to lan66845_07_c07_p191-228.indd 215 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions measure one’s level of response set social desirability, such as the Marlowe-Crowne Social Desirability Scale (Crowne & Marlow, 1960; Marlow & Crowne, 1961) and the lie subscale of the MMPI-2 (Pearson Education, Inc., 2007). Interestingly, the MMPI-2 also has a fake bad subscale as well as a lie subscale. One other common type of within-subjects artifact is known as participants’ self-per – ception, which occurs when the participants change themselves (on their own) during the course of the study. In many cases, the experimenter wants an assessment of current behavior (although sometimes the goal of a study is, indeed, to change a participant’s behavior). However, this within-subjects artifact occurs when participants decide for themselves to change their own behavior, and this behavior change is not a planned part of the study. A classic example of this occurring comes from the industrial/organizational psychology literature, where assembly line workers placed in a special situation banded together to work hard to impress the researchers (Roethlisberger & Dickson, 1939). The details of this classic study are presented in this chapter. Demand Characteristics Demand characteristics, first introduced in Chapter 3, are another data collection artifact, stemming from the participants’ understanding of what the experiment is all about, and potentially responding the way the experimenter wants them to (in a manner of speaking, giving in to the demands or expectations of the experimenter). One method of dealing with this is to disguise the nature of the study so that the participant has difficulty discerning the hypothesis and giving the experimenter what he or she is looking for. Along those lines, the partic – ipants could be uninformed about the complete nature of the study and not told about it until the study’s conclusion. This approach is called a single-blind study because the participants are “blind” to (that is, they do not know) the condi – tion of the experiment they are participating in (this has nothing to do with visual abilities, and this term may be considered offensive by some). You should note, however, that this involves the use of deception, and such steps should be considered at length (we mentioned the pitfalls of deception in Chapter 2). Often it is sufficient that the participants know in general about the study, but they do not know what specific con – dition or group they are in, hence not knowing how to respond to a demand characteristic. If participants cannot ascertain whether they are in the experimental or control group, then a single- blind study is under way and the demand char – acteristics can be minimized. One method to determine if the independent variable manipula – tion worked is to simply ask participants about it in a post-experimental interview. In a single-blind study, participants do not know all the conditions of the experiment they are participating in. RubberBall/SuperStock lan66845_07_c07_p191-228.indd 216 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions Experimenter Expectancy One additional type of data collection artifact is experimenter expectancy. This bias occurs because the experimenter (in this case, the person conducting the experimental session) accidentally influences the participants to perform in a certain, unnatural manner. This might happen if two different experimenters were used for the experimental or control groups, different instructions were used, or if one experimenter was very friendly to the experimental group but cold to the control group. To avoid these effects, the experiment could be performed (run) in one session if feasible, experimenters can be trained to avoid experimenter expectancy cues, or a double-blind study can be performed. In a double- blind study, neither the participants nor the experimenter in the room know which par – ticipants are in which group (experimental or control). In this case, the experimenter cannot unknowingly provide performance cues (i.e., expectations) to the participants, because the experimenter does not know which group is which. Someone else helping to administer the experiment knows of the group assignments and reveals them only when the data collection segment of the experiment is over. For more of the classic work on experimenter expectancy, see Rosenthal’s work (1966; 1967). Pilot Testing Your Study Think of a pilot test or pre-test as a dress rehearsal prior to conducting your study. It is wise to pilot test, because in measuring human behavior, elements of an experiment can go wrong if details are not attended to. For example, in survey research, a pilot test can help you to determine if participants understand your survey questions and if you are completely covering the topic as you intended (Collins, 2003). There are typically four goals to achieve when pilot testing your survey prior to launch. The survey researchers want to evaluate the draft survey items, optimize the length of the scale for adequate response rate, detect any weaknesses in the survey, and attempt to duplicate the condi – tions under which the survey will be administered. In a study asking college students about health risk-taking behavior Daley, McDermott, McCormack Brown, & Kittleson (2003) effectively used multiple rounds of pilot testing before wide distribution of multiple web-based surveys. Pilot tests indicated, for example, that the time to completion was 22 minutes, a 75% response rate, and students believed that the web interfaces were poorly designed. With this information gleaned from the pilot testing, Daley et al. (2003) were able to make changes to the design of the survey prior to launching a data collection effort aimed at over 1,500 college students. When designing survey research, you may want to ensure respondents (1) know the answers, (2) can recall the answers, (3) understand the questions, and (4) are comfort – able reporting the answers in the survey context. For instance, make sure that the survey items are at a reading level that is appropriate for first-year college students. Additionally, when you are collecting data with a Likert-type agreement scale (strongly disagree, disagree, neutral, agree, strongly agree ), the survey items should be declarative sentences, and not phrased in the form of a question. By assuring participants that their data are anonymous, and not linking their identity to responses, you encourage honesty about sensitive topics or illegal behaviors. Pilot testing allows you to find most problems that may occur in your study before conducting your study. Here are some quick reminders for you to consider before your pilot testing phase (from Litwin, 1995): lan66845_07_c07_p191-228.indd 217 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions • Are there any typographical errors? • Are there any misspelled words? • Does the item numbering make sense? • Is the font size big enough to be easily read (on paper, on the screen)? • Is the vocabulary appropriate for the respondents? • Is the survey too long? • Is the style of the items too monotonous? • Are there easy questions mixed in with the difficult questions? • Are the skip patterns too difficult to follow? • Does the survey format flow? • Are the items appropriate for the respondents? • Are the items sensitive to possible cultural barriers? • Is the survey in the best language for the respondents? Manipulation Checks and the Post-Experiment Interviews In some research scenarios, the independent variable involves a manipulation where the participant is intended to undergo some temporary change in state. For instance, when peo – ple are slightly depressed, how do they react when listening to music that has lyrics that are remorseful? A researcher who is testing normal, healthy volunteers may attempt to induce a moderate degree of sadness in his or her participants prior to the exposure to the lyrics. A manipulation check is a methodological procedure that occurs toward the end of the study (sometimes during a post-experiment interview) where the researcher ascertains just how sad (or not) the participant became during the study. In other words, did the intended effect of the independent variable occur? Manipulation checks may be fairly common in certain types of research. For example, in research published in the Journal of Personality in the 1980s and 1990s, over 50% of the published articles included manipulation checks (Mallon, King – sley, Affleck, & Tennen, 1998). A manipulation check can be used just to see if the indepen – dent variable manipulation worked (e.g., Keller & Bless, 2005), or a score can be generated for the level of success of the independent variable manipulation, and this score can be used as a mediating variable for further analysis (that is, the strength of the independent variable can be used statistically to help explain the outcomes on the dependent variables). A post-experiment interview may not necessarily involve a manipulation check. The post-experiment interview is precisely what it says—after the experiment is complete, the researcher interviews the participant to get an idea about the participant’s perceptions of the research experience. Did he or she understand the task? Did the debriefing provide enough information? A manipulation check may also occur during this sequence. Say, for example, that a researcher wanted to understand the impact of a happy or sad mood on completing an instructor ’s course evaluation. Course evaluations are one important com – ponent for evaluating an instructor ’s teaching effectiveness at the end of a course—more important than students may realize. So, if the student is happy or sad at the moment of the evaluation, does that impact teaching evaluation scores? A researcher in the laboratory may attempt to induce a happy or sad state in a group of participants, and then ask them to complete a teaching evaluation. The manipulation check will attempt to confirm if the participant became happy or sad. Kidd (1977) reminded us that “valid manipulation check measures may be obtainable only in certain types of circumstances, namely those in which the subjects have the opportunity to reflect and report on their psychological state and are lan66845_07_c07_p191-228.indd 218 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions willing to do” (p. 96). Thus, manipulation checks during the post-experiment interview may not always be necessary—it depends on the type of independent variable being used. Data Collection and Storage At first, the notion of data collection appears straightforward: In this part of the research, you collect your data. However, like most processes, it’s much more complicated than that. Thomas and Selthon (2003) described the steps that are involved in data collec – tion: (a) plan for the data collection process; (b) test data collection procedures in a pilot test (presented earlier); (c) collect data; (d) code the data for further data analysis (could involve creating a codebook; see more below); and (e) edit the data—check for accuracy (addressing issues such as missing data and outliers). There are many methods of data collection available to researchers, and a comprehen – sive review is not possible here. But for each of these variations, you are likely to find expert advice, whether it be for collecting data on the Internet (Birnbaum, 2004; Cantrell & Lupinacci, 2007; Courtney & Craven, 2005), or collecting and storing qualitative data (Levine, 1985), for example. Regardless of your approach, you planned for data collection, you conducted your pilot tests, and you collected your data—now it’s time to prepare for data analysis. It’s time to explore the data that you have. You’ll hear different terms for this, such as data screening (Pryjmachuk & Richards, 2007) or data verification (Thomas & Selthon, 2003). Pryjmachuk and Richards (2007) advised “caution demands that, prior to full data analysis, researchers employ procedures such as data cleaning, data screening, and exploratory data analysis” (p. 43). By making sure, as much as possible, that your data are accurate, you help to ensure the integrity of your results. Said another way, if you based your statistical analysis and research conclusions on faulty data, then the conclu- sions themselves are faulty. It is important to check to see that the data the participants provided was entered and coded correctly. When discussing data cleaning, two frequent topics emerge that warrant our attention here—outliers and missing data. There are different ways that we can think about outliers. Some can be data entry mis – takes, or others can be implausible entries that could be correct but have a high likeli – hood of being incorrect. Data entry mistakes are sometimes easy to find. For instance, consider the results for a survey using a response scale from 1 = not at all confident to 5 = extremely confident . If you were looking at this type of data, and you saw an entry of 0 or an entry of 6, then you would know that this was an error. Ideally, you could go back to the original survey and replace this value with the correct answer. Sometimes you might see a 22 or 34, which is a data-keying error. Perhaps the actual answer was 2 (for the first example)—but for the second example, was the actual response a 3 or a 4? Ideally, you’d go back to the original survey and correct the mistake. If the original data were not available to you, you’d probably delete that particular observation—more on missing data in a moment. Some outliers are easy to identify, (0, 6, 22, 34), and there are multiple fixes available. Outliers typically arise from two sources, coding errors or two different populations within the sample (DiLalla & Dollinger, 2006). Coding errors need to be corrected, either by checking against the original data or deleting that data point (more options exist— more on that momentarily). Having two different populations in your sample is more lan66845_07_c07_p191-228.indd 219 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions difficult to ascertain. DiLalla and Dollinger (2006) provide an example where you might be testing, in a sample, some individuals who are psycholog – ically healthy and others who suffer from a men – tal disorder. Ideally, you would have the ability to separate these two populations and then conduct separate statistical tests on the two samples. You can only do this, however, if you suspect ahead of time that you will be capturing two or more popu – lations and you have a reliable and valid measure to allow you to separate the populations after the data are collected. It is important to deal with out – liers because they can wreak havoc on your statis – tical conclusions (Pryjmachuk & Richards, 2007), particularly if your sample is relatively small. Dealing with missing data can be equally com – plex. You may give instructions to participants that they should leave blank any questions that they do not want to answer. Or, you instruct par – ticipants to answer every question, but they don’t follow instructions. What do you do when the data are missing? As a student, if you are in this situation in a course (or working with a faculty member as a research assistant), your instructor or supervisor will have some guidelines for you. One conservative approach would be to not use data from a participant unless the data are complete, that is, none are missing. However, we often instruct participants not to answer questions that don’t apply to them, so in many cases the absence of data indicates that the participant was following instructions. You could set a priori (meaning before the fact) a level of acceptable missing data. DiLalla and Dollinger (2006) report that in personality research using surveys, a common threshold is 5% missing data. Thus, if you are asking 100 survey items on a questionnaire, you would keep a participants’ data if they left blank up to 5 items, but if 6 or more items were left blank, your a priori rule states that you would not include that person’s data in your data set. Handling decisions about missing data in this way makes much more sense if you set the decision rule prior to the study. A final consideration in the data collection process is data storage. Although this may seem straightforward, it is an important consideration. You’ll want to keep original sur – veys or files that contain data throughout your project. If you are collecting data with the eventual intention of publishing that research, you’ll need to keep and archive your data for at least 5 years. Not only will you need to keep the originals, but make sure you keep (and back up) your electronic files, such as the SPSS data file and your codebook. You may need to think about a storage plan as well. For instance, when you apply for IRB approval, part of the application asks you about where the data will be stored, and how it will be secured. Depending on the type of research you are conducting, your data may be linked to individuals, and you’ll want to be sure that you take steps to ensure anonym – ity and confidentiality. Thus, if you are keeping paper files, where will these be located? Will they be stored in a locked filing cabinet? For electronic files, will these be stored on a computer, multiple computers, or USB memory stick? If so, who will have access—will Outliers can skew data if there are coding errors or if the sample used to collect data came from two different populations. Exactostock/SuperStock lan66845_07_c07_p191-228.indd 220 4/20/12 2:50 PM CHAPTER 7 Section 7.6 Procedural Matters: Experimental Pitfalls and Precautions the individual files be password protected in case someone finds your data files? There are steps you can take to store your data anonymously, but if that is not possible or plausible, you’ll need to carefully consider a data storage plan. Once you’ve completed your data collection process, and you are as confident as you can be that your data are clean (i.e., accurate), you’ll be ready for statistical analysis. But the type of data you glean from psychological research very much depends on how the research is designed. That is, the type of research you design will substantially influence both your data collection and data analysis options. There are research basics inherent in all good research designs. The incorporation of the information in this chapter along with previous chapters will help to improve your research efforts and ultimately lead you to better and more meaningful conclusions from the data collected. Case Study: Precision Matters (Careful: One Size Rarely Fits All) The entire premise of this chapter is that observation practices and measurement operations matter. They truly do. If psychology students are not careful to apply critical thinking skills learned throughout their undergraduate education, we can be as gullible as the average citizen. Thus, the measures that we use to capture behavior—dependent variables—need to be meaningful if the results from our studies and projects are to have any impact on the discipline. Testing your hypothesis within the con – text of an applied project means that your study goes out on a limb (so to speak) to make a prediction that could be supported or refuted. That is, there needs to be specificity regarding the hypothesis and what the eventual outcomes would look like if supported or refuted. But what if your hypothesis were so vague or broad that it would be hard to refute? If it were too vague or too broad, then the outcomes would not have much meaning—sometimes a one-size-fits-all approach results in lackluster outcomes. For example, if your hypothesis were something like “tomor – row the sun will rise in the east and set in the west,” well, that’s not much of a hypothesis. But what if the setup to the hypothesis were not so obvious? Say, for example, you are in a study where the research is interested in identifying personality characteristics of the general population. You’ve just completed a 25-item personality inventory (online), and 10 minutes later you receive an email with your results. Based on the analysis of your data, your feedback looks like this: “You have a great need for other people to like and admire you. You have a tendency to be critical of yourself. You have a great deal of unused capacity that you have not turned to your advantage. While you have some personality weaknesses, you are generally able to compensate for them. Disciplined and self-controlled outside, you tend to be worrisome and insecure inside. At times you have serious doubts as to whether you have made the right decision or done the right thing. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limita- tions. You pride yourself as an independent thinker and do not accept others’ statements without sat – isfactory proof. You have found it unwise to be too frank in revealing yourself to others. At times you are extroverted, affable, and sociable, while at other times you are introverted, wary, reserved. Some of your aspirations tend to be pretty unrealistic. Security is one of your major goals in life.” Most people would tend to agree with this assessment of their personality, precisely because it is so broad and vague! At times this is called the Barnum effect in psychology (after the circus showman P. T. Barnum who frequently promised “something for everyone”); the demonstration originated with Forer (1949). Often we are gullible when it comes to interpreting information about ourselves, and we may not apply the same critical thinking approach to ourselves as we do to the study of others. Whitbourne (2010) wrote about this topic from the perspective of fulfillment. She suggested that we may be gullible in these types of situations because (1) the message is so broad that there literally is something that applies to everyone in such a vague statement; (2) we welcome comforting predictions about the future because the unknown aspect of the future is scary to some; and (3) we are motivated to (continued) lan66845_07_c07_p191-228.indd 221 4/20/12 2:50 PM CHAPTER 7 Section 7.7 Causality and Drawing Conclusions from Evidence 7.7 Causality and Drawing Conclusions from Evidence T he most powerful conclu – sion that we can make using science is a cause- and-effect conclusion. To be able to determine the causality of events is powerful, because in theory we could make posi – tive outcomes occur more often and work to prevent negative outcomes from happening as often. For example, it would be beneficial to know what causes marital satisfaction, what causes happiness, what causes col- lege student success, and what causes self-actualization so that we could promote those causes want to believe statements about our own personality, so we read more into the vague statements than usual, searching for nuggets of truth. The same broad, vague approach that applies to personality statements can also apply to situations where you want to believe what is on your fortune cookie or an astrology reading or daily horoscope (Ward & Grasha, 1986). Careful observation and measurement can assist in the psychological myth- busting of being gullible and believing in such broad, vague statements. There are risks involved in believing in such myths (Whitbourne, 2010), such as potentially wasting your money, being given poor advice, and ignoring good advice because it lacks the entertainment value of a horoscope or an astrology reading. Assessments that are based on solid science, applying the fundamental principles of observation and measurement, are likely to be much more accurate and predictive of your future than Barnum effect-type statements that are so vague that they nearly fit all. Reflection Questions 1. Do you read your horoscope? How often? Do you read it for fun, or are there times when you fol – low the advice given on a particular day? Did you ever follow the advice and discover that it led to a good outcome or a bad outcome? What benefit might there be in applying a scientific approach to recording and measuring systematically the successes and failures of your horoscope readings? 2. What are the types of situations in life where you may be more susceptible to persuasion and influence regarding decisions involving scenarios where a description fits you to a “T”? Ever gone shopping for a new car (or at least a car that is new to you)? What about shopping on Craigslist or eBay? Are there other scenarios besides shopping where you might be more gullible to believe what someone is saying about you? 3. As you think about and reflect upon the applied project that you have designed, what are the key observation and measurement components of your study? What is your dependent variable (or what are your dependent variables)? Are they measured in such a way that precise measure – ments are possible and a viable test of your hypotheses can occur? Have you avoided a “one-size- fits-all” scenario? How so? Case Study: Precision Matters (Careful: One Size Rarely Fits All) (continued) Determining cause and effect is one of the most powerful and difficult conclusions achievable in science. Photodisc/Thinkstock lan66845_07_c07_p191-228.indd 222 4/20/12 2:51 PM CHAPTER 7 Section 7.7 Causality and Drawing Conclusions from Evidence and help individuals strive toward achieving their goals. Conversely, it would be nice to know what causes Alzheimer ’s disease, what causes autism, what causes clinical depres – sion, what causes low-self esteem, and what causes suicide so that we could work to pre – vent antecedent (before-the-fact) causes that lead to these negative outcomes. However, it takes precise methodology to arrive at any level of confidence about causality, and there are many different forms of research questions to ask. Meltzoff (1998) does a very nice job of describing the types of research questions. Meltzoff ’s description is summarized in Table 7.4, using generic statements but with realistic examples. Table 7.4: Types of research questions, with examples Types of Research QuestionsGeneric ExampleSpecific Example Existence Questions Does x exist?Can people have a Facebook addiction? Does sincere altruism exist? Questions of Description and Classification What is x like? To what extent does x exist? What are the best practices of master teachers? What is graduate school like? To what extent are teacher-created tests like the GRE? Questions of Composition What are the components that make up x? What are the factors that make up x ?What variables lead to high student satisfaction with college? What are the leading indicators that someone is clinically depressed? Statistical Relationship Questions Is there an association or relationship between x and y? Is one’s age related to GPA? Is there an association between gender and political affiliation? Descriptive-Comparative Questions Is Group x different from Group y? Are males or females more likely to stay in college? For adults returning to school, do parents or non-parents have a better GPA in college? Causality Questions Does x cause, lead to, or prevent changes in y? Does psychotherapy help individuals with dissociative identity disorder? Does attending tutoring sessions lead to better student test performance? Causality-Comparative Questions Does x cause more of a change in y than z does? Is Prozac better than Xanax at helping people deal with depressive symptoms? Does caffeine help with better concentration skills as compared to a placebo? Causality-Comparative Interaction Questions Does x cause more change in y than does z under certain conditions but not under others? Are male Republicans more likely than female Republicans to vote for a Democratic nominee? Are psychology majors more likely to be successful in their health care- based careers than non-majors, but only for psychology majors who attend graduate school? For many scientists, the ultimate goal is the determination of causality; that is, understand – ing cause-and-effect relationships. There are three criteria for establishing causality, as lan66845_07_c07_p191-228.indd 223 4/20/12 2:51 PM CHAPTER 7 Section 7.8 Proving Versus Disproving in Psychology summarized by Burns (1997). First, there must be clear temporal precedence . This means that for the cause to be the cause, and the effect to be the effect, the cause must come first and the effect must come second. If both cause and effect occur simultaneously, then we cannot know the cause or the effect. There must be a clear time sequence here. Second, measures of cause and effect must covary. That is, if there is a cause-and-effect relation- ship, the presentation of the cause needs to yield the effect, but if there is no presentation of the cause, then there should be no effect. If you change the nature of the cause, then you should also be changing the nature of the effect. Lastly, there should be no plausible alter – native explanation . If we have adequately applied our research methods, experimental controls, methodological designs, and so forth, then we need to say, with confidence, that there is no other logical explanation for the effect other than the cause. Note that we do not say that we proved that the cause is the reason for the effect, but we infer the relation – ship when we (a) have temporal precedence, (b) have covariation, and (c) have ruled out plausible alternative explanations (Burns, 1997). Technically speaking, we don’t “prove” anything in psychology, but we disprove. 7.8 Proving Versus Disproving in Psychology T he notion of falsificationism is important to science and psychology. A key contribu – tor to this notion was Karl Popper, who suggested that the goal of science should not be to confirm or prove theories, but to falsify or disprove theories. The approach taken by Popper and others is not merely semantic double-talk, but has serious methodolog – ical implications for how we carry out science and how we advance our knowledge of the human condition. Newell (2005, para. 3–5) summa – rizes this position nicely when he gives an example of the falsification approach if we were to test the proposition ‘all swans are white.’ This can never be proven, since that would require checking each and every one of them everywhere; but it can be dis proven by finding a single instance of a non-white swan. A the – ory is scientific, then, if we can say what would possibly cause us to reject it. Although a theory is never proven, if we can falsify it then we force our – selves to look again and come up with a better one. (italics in original) Researchers develop a general theory and then generate a number of plausible alternative expla – nations or hypotheses that would defeat or dis – prove the theory. If the alternative explanations turn out to be correct, then the theory lacks sup – port. If the alternative explanations are not sup – ported, then the theory is still alive and well. We begin with a general idea and numerous According to Karl Popper, the purpose of science is to falsify or disprove scientific theory. Do you agree with Popper? Associated Press lan66845_07_c07_p191-228.indd 224 4/20/12 2:51 PM CHAPTER 7 Concept Check alternative explanations; our goal is to disprove the alternative explanations so that the only rational idea left standing is our theory (sort of a “king of the hill” situation, but with ideas). That is how psychological theories are supported—by disproof, not proof. As psychologists-in-training, be careful with the language you use. Sometimes students want to be able to say that they “proved” something, particularly when writing the Discussion section. Remember, we don’t prove anything, but we attempt to disprove competing theo – ries until the only plausible explanation left standing is our working hypothesis. Chapter Summary T he principles of the scientific approach in psychology—observation and measure – ment—are presented in this chapter. In any experiment or quasi-experiment, basic fundamental decisions have to be made concerning independent and dependent variables. For dependent variables, how will they be measured, and if measured quan – titatively (which is frequently the case in psychology), on what scale will they be mea – sured? What operations will be followed to ensure reliability and validity of the data gathered and the conclusions drawn? Once the foundational questions are answered, then a plethora of practical matters must be considered, such as avoiding confounding variables, avoiding data collection artifacts (and threats to validity), pilot testing, manip – ulation checks, and data collection and storage. With all the care applied to every step of the process, meticulous decisions are made based on the outcomes of the study, taking care not to draw conclusions that are overzealous and/or not supported by the data pre – sented. The complexities and intricacies of this process are just some of the reasons why advanced training is needed—such as the undergraduate degree—so that psychological finding can be properly reported and utilized in an applied manner to help improve the human condition. Concept Check 1. Which of the following would NOT be an example of a variable? A. 10 years old B. Gender C. Breed of dog D. Shoe size 2. Which of the following would NOT be a quantitative variable? A. Ounces of liquid B. Speed of completion C. Genre of book D. Number of items answered correctly 3. Classical test theory claims that a measurement is the A. sum of knowledge and experience. B. sum of true score and error. C. difference between ability and aptitude. D. difference between right and wrong answers. lan66845_07_c07_p191-228.indd 225 4/20/12 2:51 PM CHAPTER 7 Key Terms to Remember 4. The split-half method of reliability is a form of A. test-retest reliability. B. internal consistency. C. interrater reliability. D. alternate forms reliability. 5. A priori analysis refers to analysis A. done on more than two groups. B. executed before data collection. C. completed before other analyses. D. planned before data collection. Answers 1. A. 10 years old. The answer can be found Section 7.1. 2. C. Genre of book. The answer can be found Section 7.1. 3. B. Sum of true score and error. The answer can be found Section 7.3. 4. B. Internal consistency. The answer can be found in Section 7.3. 5. D. Planned before data collection. The answer can be found in Section 7.6. Questions for Critical Thinking 1. Think about the perceptions you had about psychology before you began your formal, college-level study of psychology? Did you think that psychology would be easy compared to some of the other disciplines you might have studied? How do you think about psychology now? Is it as easy as you once thought? Which components of an education in psychology are you finding the most worthwhile, and which components seem disconnected from other avenues of study you are pursuing? 2. You have completed a number of courses in different disciplines, and probably other courses in the social sciences outside of psychology (sociology, criminal justice, anthropology, economics, and so on). How does a psychological approach to studying human behavior differ from the approaches of other social sciences in studying human behavior? To what extent are these principles of observation and measurement similar to or different from the approaches in other social sci – ence disciplines? Key Terms to Remember alternate forms A test where a researcher develops two different forms of a test that are designed to be parallel but do not meet the same criteria levels for parallel forms. artifact When the measurement process is distorted, biased, or corrupted in some fashion. coefficient of equivalence The correla- tion coefficient that results from a parallel forms test. See parallel forms. coefficient of stability A correlation coef- ficient that results from testing and retest – ing a score over time. lan66845_07_c07_p191-228.indd 226 4/20/12 2:51 PM CHAPTER 7 Key Terms to Remember concurrent validity The assessment of how the score on a test or inventory is related to your current state of affairs. confound An event or occurrence that happens at the same time of your study that is not part of your designed study but can influence its outcome. construct validity When a test measures what it purports to measure. Also known as “umbrella validity.” content validity The determination as to whether or not the composition of items that make up a test reflects the universe of ideas, behaviors, and attitudes that com – pose the behavior of interest. covary To establish temporal precedence in a cause-and-effect relationship, the effect must be evident upon presentation of the cause, If there is no presentation of the cause, then there should be no effect. See temporal precedence. criterion-related validity The assessment of how the measurement outcome, or score, relates to other types of scores. external validity The assessment of whether or not a causal relationship can be generalized to other research settings, samples, or times in the event that a causal relationship has been determined to exist between the independent and dependent variables. face validity The assessment of whether or not the person taking the test believes that the test is measuring what is purports to measure. falsificationism The concept that the goal of science should not be to confirm or prove theories but rather to falsify or disprove theories. internal validity The assessment of the general nature of the relationship between the independent variables and the depen – dent variables. It primarily focuses on the determination of causality and whether or not the manipulation of the independent variables caused changes in the dependent variables. interrater reliability A method of deter – mining reliability in which two or more raters categorize nominal data and obtain the same result when using the same instrument to measure a concept. interval/ratio An interval scale presents numbers in a meaningful way and pro – vides equal intervals including zero. In a ratio scale, numbers are used in the typical manner, where 0 = a lack of something. The two scales of measurement are usually combined in psychological research since their interpretation individually can pres – ent challenges. measurement How the responses of indi- viduals are captured for the purposes of research. operational definition A concise defini – tion that exhibits precisely what is being measured. parallel forms A test where a researcher administers two versions of a test to the same group of individuals, resulting in a correlation of the outcomes between the two test administrations. See coefficient of equivalence. pilot test A “practice run” of a ques – tionnaire used to determine weaknesses and optimize the length of the scale for adequate response rate. The conditions in which the survey will be administered are typically replicated as closely as possible to the actual survey administration. lan66845_07_c07_p191-228.indd 227 4/20/12 2:51 PM CHAPTER 7 Web Resources predictive validity When a researcher takes current knowledge and attempts to make a prediction about the future. plausible alternative explanation The ability to state, with confidence, that there is no other logical explanation for the effect other than the cause. qualitative variable A variable in which the responses differ in kind or type. quantitative variables Variables that are measured on a numeric or quantitative scale. response set A pattern of responding seen in a participant that may not accurately reflect the participant’s true feelings on a topic. response set acquiescence When participants get stuck in the trend of responding yes repeatedly in a survey or questionnaire. response set social desirability When participants respond in a pattern that they believe makes them look good, or look bet – ter than they are. scales of measurement Tools used to translate observations into scores in nomi – nal, ordinal, interval, or ratio scales. split-half method A method of estimating internal consistency that involves splitting the instrument in half and then correlating the scores from the resulting halves. statistical conclusion validity The assessment of whether or not method – ological and statistical approaches used in an experimental situation are sensitive enough to capture a causal relationship. temporal precedence To determine what is the cause and what is the effect, the cause must come first and the effect must come second. If they occur at the same point in time, then the determination of which is the cause and which is the effect cannot be made. validity The determination as to whether or not researchers are truly “measuring what they think they are measuring” for the purposes of their research. variable An entity that can take on differ – ent values. Web Resources This website provides a video that gives a short lesson on dependency relationships and establishes the differences between dependent and independent variables. ht t p://w w w.yo ut u b e.c o m/wat c h? v=ut Np S E EyM I U This website provides definitions, examples, and in-depth interpretation of validity and establishes the differences between different types of validity. http://writing.colostate.edu/guides/research/relval/pop2b.cfm This website outlines how to collect data, including a brief introduction to the process of getting approved by an Institutional Review Board and obtaining informed consent. http://www.socialpsychology.org/consent.htm This website explains levels of measurement and the best way to apply them in a research setting through defining types of scales and explaining when it is appropriate to use specific scales. http://cnx.org/content/m10809/latest/ lan66845_07_c07_p191-228.indd 228 4/20/12 2:51 PM 8 Applying Psychology: To Workplace, to Life Chapter Learning Outcomes After reading and studying this chapter, students should be able to: • comprehend the importance of networking in psychology and being active in the field, including attending conferences and reading widely published works about human behavior. • appreciate the high value of undergraduate research and know that many benefits can accrue from involvement in research, including the establishment of a mentoring relationship with a faculty member. • recognize the importance of national-level organizations to help organize and coalesce the broad field of psychology into meaningful and value-added organizations such as AP A, APS, and Psi Chi. • describe basic graduate school admission strategies and know the next steps to be taken if a student wanted to pursue this post-baccalaureate opportunity. • describe the basic transitions processes from college to career and recognize the potential pitfalls and behaviors that can get a new college hire demoted or fired, as well as know the behaviors that can lead to hiring and promotion in the workplace. • reflect on their psychology major as well as aspirational goals, whether related to a career or graduate school, and understand some of the next steps to be taken after self-reflection and career planning. • describe what it means to think like a psychologist, and to comprehend the basic, fundamental beliefs of scientists trained in psychology and their accompanying views of the world. Goodshoot/Thinkstock lan66845_08_c08_p229-258.indd 229 4/20/12 2:51 PM CHAPTER 8 Introduction Introduction A s an undergraduate, it’s easy to think of psychology as this very static discipline, and if you want more information about some type of behavior, you conduct a search and the information comes to you. As you fulfill the curriculum of your undergraduate program, your professors and your online courses bring you information, and your textbooks provide a wealth of knowledge about the subject matter. The Voices from the Workplace feature box describes a passive approach to learning and under – standing human behavior. Here I would encourage you to take a more active learning approach—that is, if you want to get a sense of what psychology is all about, you have to go and do psychology. We belong to an active and engaging discipline that is passionate about all aspects of human behavior, and although we do share knowledge in various forms of writing (journal articles, books, websites), interacting with peers and profession – als in a conference setting can provide the energy and “juice” about the research enter – prise. So I suggest that you go and do psychology: Work to become an active contributor to our understanding of human behavior as well as a consumer of psychological knowledge. Voices from the Workplace Your name: Steve S. Your age: 37 Your gender: Male Your primary job title: President & CEO Your current employer: Solera Networks How long have you been employed in your present position? 10 months What year did you graduate with your bachelor’s degree in psychology? 1992 Describe your major job duties and responsibilities. Responsible for the day-to-day operations and strategic positioning for a high-tech startup. What elements of your undergraduate training in psychology do you use in your work? 1. Interpersonal relationship skills. 2. Pattern recognition. 3. Positive and negative reinforcement techniques. What do you like most about your job? Each day presents a new and unique set of challenges. I enjoy rallying a team of smart people around a goal and driving the company to achieve that goal. What do you like least about your job? “Administrivia”—I really dislike the tactical administration aspects; things that most of us take for granted in larger corporations. Beyond your bachelor’s degree, what additional education and/or specialized training have you received? Two years of graduate school in psychology. (continued) lan66845_08_c08_p229-258.indd 230 4/20/12 2:51 PM CHAPTER 8 Section 8.1 Doing Psychology 8.1 Doing Psychology G ood scientists communicate openly and make knowledge public. It does science and psychology no good to conduct wonderful, empirical, data-driven studies just to have the results end up in a folder in a filing cabinet or stored as a file on a hard drive. Good science shares the details of scientific discovery publicly. So, to “go do psychology” means to share your results and your research findings with a larger audience. That audience could be an on-campus undergraduate research and scholarship What is the compensation package for an entry-level position in your occupation? In a CEO position for a startup, one should expect a lower minimum salary and more equity. Base sala- ries are quite varied but something starting in the 150K range seems reasonable. What benefits (e.g., health insurance, pension, etc.) are typically available for someone in your profession? Health, Dental, and Life are typical. In a startup, one should not expect a 401k; rather company equity is more common. What are the key skills necessary for you to succeed in your career? 1. Work ethic—being willing to put in the hours to insure success. 2. Attention to detail. 3. Good pat – tern recognition and interpersonal skills. Thinking back to your undergraduate career, what courses would you recommend that you believe are key to success in your type of career? Intro to Learning, Statistics, Social Psych. Thinking back to your undergraduate career, can you think of outside of class activities (e.g., research assistantships, internships, Psi Chi, etc.) that were key to success in your type of career? Working in Hal Miller’s lab was a great experience. Acting as a manager of the operation gave me great experience in handling budgets, people, and projects. Looking back, this may have provided the single most salient experience that could be directly applied to my current position. As an undergraduate, do you wish you had done anything differently? If so, what? I would have changed my minor to be more practical (from Analytic Philosophy to Economics). I may have focused more on applied psychology versus basic research. What advice would you give to someone who was thinking about entering the field you are in? Be prepared to work hard, face seemingly insurmountable obstacles with zeal, and otherwise be pre – sented with daunting challenges. The payoff is satisfaction in growing something from nothing, and mak – ing a “mark” in your representative industry. Ultimately, if one is successful, financial reward will follow. If you were choosing a career and occupation all over again, what (if anything) would you do differently? I may have gone on to law school instead of graduate school, but I ultimately enjoy what I am doing now very much. Copyright © 2009 by the American Psychological Association. Reproduced with permission. The official citation that should be used in referencing this material is R. Eric Landrum, Finding Jobs With a Psychol- ogy Bachelor’s Degree: Expert Advice for Launching Your Career , American Psychological Association, 2009. The use of this information does not imply endorsement by the publisher. No further reproduction or distribution is permitted without written permission from the American Psychological Association. Voices from the Workplace (continued) lan66845_08_c08_p229-258.indd 231 4/20/12 2:51 PM CHAPTER 8 Section 8.1 Doing Psychology conference; a local conference hosted by your Psi Chi chapter or department of psychol – ogy; a statewide, regional, or national meeting of psychologists that includes student work; or perhaps a publication in a student journal. Yes, there are conference opportu – nities and journal publishing opportunities especially designed for students—these are opportunities for professionalization into the discipline, so that you can see what it is like to be a psychologist and to make contributions like psychologists do. Professionalization in Psychology: Research, Conferences, Publications Empirical research is at the heart of psychology, which is why your applied project course is so important. Perhaps all you want to do with your psychology major is to “help peo – ple,” and you may not immediately understand why all this science and research stuff is so important. Think about it: We are an evidence-based discipline, so if you were a therapist or psychosocial rehabilitation worker, wouldn’t you want to know that what you were doing was helping? That is, counselors and clinicians may not always be active researchers, but they will always be consumers of research. You may not want to continue to do research after receiving your bachelor ’s degree in psychology, but you will need to be able to read, understand, and interpret research—and that is one reason why there is so much empha – sis on research methods in psy – chology. The entire research process—from idea conceptu – alization to literature review to research design to pilot testing to data collection and analysis to statistical reporting and report writing—these are all key skills and abilities that bode well for your future, whether you are going to graduate school or not. So even if your future goal is to help people (which is a very noble goal), you’d want to help with the best and most efficient means possible. And to know that , you’ll need to be able to comprehend published research studies. This emphasis on research in psychology sometimes leads students to think about the teaching versus research dichotomy and ask, “Which is more important?” The answer I would give would be that both are necessary, and neither is more important. Without research, there would be very little for teachers to teach, and without teachers, no one would ever learn how to conduct research. The research-based experiences you have had in your psychology classes up to and including now should be preparing you for continued research experience, such as serv – ing as a research assistant for a faculty member, completing an internship or independent study, or perhaps conducting a senior thesis project. No matter what your future goals While counselors and clinicians may not conduct research once they enter their field, they often apply information drawn from ongoing studies throughout their career. Phanie/SuperStock lan66845_08_c08_p229-258.indd 232 4/20/12 2:51 PM CHAPTER 8 Section 8.1 Doing Psychology after obtaining your bachelor ’s degree in psychology, try to find opportunities during your undergraduate career to become actively engaged in a program of research with a faculty member. Not only is this a great way to see the practical application of what you have learned, but it also gives you an opportunity to perhaps build a mentoring relation – ship with a faculty member, who can be very helpful as a future job reference or letter of recommendation writer for graduate school. In addition to becoming more involved in research endeavors before you graduate, you should know that there are multiple outlets specifically designed for undergraduate work. For instance, there are dedicated journals specifically designed to publish your psycho- logical research. Table 8.1 provides a listing of five journals that are specifically designed to publish undergraduate psychology research studies. Table 8.1: Journals that publish undergraduate psychology research studies, with URLs Journal of Psychological Inquiry http://jpi.morningside.edu/ Journal of Psychology and the Behavioral Sciences http://view.fdu.edu/default.aspx?id=784 Modern Psychological Studies http://www.utc.edu/Academic/Psychology/MPS _Submissions.php Psi Chi Journal of Undergraduate Research http://www.psichi.org/pubs/journal/ You should know in advance that publishing psychological research is a lot of work. In fact, a more typical route for research (any research, not just undergraduate student research) might be to present that research at a conference first and then follow up with a publication. There are many good opportunities for undergraduate students to become involved in conference experiences, with the details to follow. Local, Regional, and National Opportunities Good science communicates through a number of venues, and writing is incredibly important in science. But psychologists communicate in other venues as well, such as local, regional, national, and international conferences where psychologists (and psychol – ogy students) make oral presentations to an audience, as well as present posters. The Internet has quickly become an important venue for sharing information about psychol – ogy, whether it is through instructor course sites, wikis, blogs, podcasts, or otherwise. As an undergraduate student, you may have the opportunity to attend a conference on your campus, or perhaps even a regional conference. Your campus may have its own multi-department annual conference. Sometimes a department of psychology will host a student research day, where only psychology stu – dents participate in the events, and there is typically a speaker who provides an invited address. You should ask some of your psychology faculty members if these opportunities are available to you locally. If not, think about starting such an effort. For example, if you are located in a region where other colleges and universities are nearby, you might think about organizing a Psychology Research Day where multiple institutions gather on one lan66845_08_c08_p229-258.indd 233 4/20/12 2:51 PM CHAPTER 8 Section 8.1 Doing Psychology campus to provide the types of conference opportunities discussed in this chapter. And why not host such an event online to spotlight the research efforts of classrooms from around the country (and perhaps the world)? Fortunately, there are a number of regional opportunities for undergraduate students to continue their involvement in the research process. In the United States, seven regional psychology associations hold annual conventions, and all include opportunities for under – graduate participation and other programming directed toward undergraduate students (portions are sometimes sponsored by Psi Chi—more on this in a bit). For more informa – tion about the regional associations, including when the next conferences are being held, submission criteria, and the like, see Table 8.2. These are all in-person conferences where travel is required, but the networking opportunities can be priceless. Table 8.2: Regional psychological associations, with URLs New England Psychological Association (NEPA)http://nepa.cloverpad.org/ Eastern Psychological Association (EPA) http://www.easternpsychological.org Midwestern Psychological Association (MPA) http://www.midwesternpsych.org Rocky Mountain Psychological Association (RMPA) http://www.rockymountainpsych.org Southeastern Psychological Association (SEPA) http://www.sepaonline.com Southwestern Psychological Association (SWPA) http://www.swpsych.org Western Psychological Association (WPA) http://www.westernpsych.org In addition to the regional conferences, there are also opportunities for undergraduates to present their research at the national level via many outlets. For instance, students give presentations at the Association for Psychological Science (http://www.psycho- logicalscience.org ) annual meeting and the American Psychological Association (h t t p : // www.apa.org ) annual convention, and these APA presentations are often affiliated with the Psi Chi (http://www.psichi.org ) program at the APA national convention. In addi – tion to these psychology-specific national opportunities, there are also more opportuni – ties for undergraduates to present research, but in multidisciplinary settings such as the Council on Undergraduate Research (CUR) annual meeting (http://www.cur.org ) and the McNair Scholars Program (http://mcnairscholars.com/ ). Although the McNair Schol – ars Program hosts a national conference, the availability of this opportunity varies by campus—a search of your own campus website should help you determine if your school has a McNair Scholars Program. So what are these conferences like, especially the regional and national conferences? That is truly difficult to describe—in some respects, you just need to go and have the experience. Sil – via, Delaney, and Marcovitch (2009) associate these academic conferences with binge think – ing—typically a 2- to 3-day experience where academics work hard to present the results of their research, and sometimes play hard too. Conferences provide a central gathering point so that the fast-moving world of researchers can catch up with one another, as well as provide valuable networking opportunities (Silvia et al., 2009). Regional and national conferences lan66845_08_c08_p229-258.indd 234 4/20/12 2:51 PM CHAPTER 8 Section 8.1 Doing Psychology also give you the chance to meet with graduate students from various schools and to interact with other undergraduates from other institutions, which can be highly rewarding as well. What would you do at these types of conferences? During the day, you’d typically listen to talks, visit poster sessions, attend workshops, check out exhibi – tors and perhaps browse or buy books, and network (Silvia et al., 2009). Thorpe and Ward (2007), in making recommendations on how to get the most out of the conference experience, sug – gested pacing yourself, wearing comfortable shoes, taking off your shy cap, and taking advantage of events specifically designed for students (at these conferences you may see Psi Chi events on the program, often centered on undergraduate students—take advantage of these opportunities if you can). Let’s say you are convinced to submit your work to a conference—now what? After you’ve found the conference you want to attend, seek out the information about submitting. Con – ferences have strict deadlines about submitting and very strict instructions about how to submit. In fact, in some of my own work with a former undergraduate student (Haines & Landrum, 2008), we asked faculty members who often review conference submissions what would be the top reasons to reject a student’s submission to a conference—in other words, what are the mistakes to avoid? About 96% of faculty indicated that poor writing quality was a reason to reject a student’s submission for conference presentation, and about 92% agreed that a late submission was a reason for rejection. Try to locate a faculty member to help ensure that your writing quality is high (by the way, some conferences will require a faculty sponsor anyway, so including a faculty member as a helper or men – tor is a good idea), and make sure you submit on time. If you are successful in getting your conference submission accepted, congratulations; but the work is not done. If you plan on presenting a poster, there are many good resources to help you with that process, including Landrum (2008), Silvia et al. (2009), Stambor (2008), and Sue and Ritter (2007). If you will be giving a conference talk, you can find some valuable tips on preparation in Landrum and Davis (2010) and Silvia et al. (2009). If you are fortunate enough to get the opportunity to attend a conference, and even more fortunate to be able to present your research, seize the opportunity and make the most of it—this is a great chance to enhance your skills and advance your professionalization into psychology. At a conference, you would either present your research as an oral paper (where you give a 12–15 minute talk about your research) or as a poster session (where you stand next to Conferences provide undergraduates the opportunity to network with graduate students and professionals from other institutions, as well as hear about the most recent research in their field. age fotostock/SuperStock lan66845_08_c08_p229-258.indd 235 4/20/12 2:51 PM CHAPTER 8 Section 8.2 The Benefits of Undergraduate Research a display that describes your research for 1–2 hours). When giving an oral paper, you are talking to an audience about the outcomes of your research. Given such a short time frame, you have to stick to the highlights, and if possible, leave 2–3 minutes at the end of your talk to answer questions or receive comments. At student presentations, audi – ence members are generally very supportive of student presenters—no one will be out to stump you, but you may be asked specific questions about your research. If you’re asked a question and don’t know the answer, be sure to say you don’t know the answer. If appro – priate, feel free to speculate about what you think, but make sure that the audience knows that you are speculating and that your opinions are just that—opinions. So how would you proceed in telling the story of your research? Luckily, the story you told in your research paper (Introduction, Method, Results, Discussion) already provides you with the structure for your talk. If possible, try not to read notes to the audience mem – bers but rather talk and interact with them. Be sure to end your part of the talk with a take- home message—if the audience is to remember only one idea from your entire research talk, be sure to emphasize that idea at the very conclusion of your presentation. If the thought of a making an oral presentation at a conference is not pleasant, there is another alternative. Poster sessions are much less stressful for most students and fac – ulty members. At a poster session you create a large-scale poster presentation of your research, and the poster is placed in a large room with other poster presenters. You stand next to your poster for 1–2 hours while conference attendees stroll past. During a poster session you’ll have many individual conversations but won’t be making a formal pre – sentation to an audience. The nice aspect of a poster presentation is that usually you get to talk to the researcher directly—in this case you are the author. Some people will walk right by; some will look at the title and keep walking; some will stop, read the poster, and not say a thing; and some will engage in conversation. Everyone has their own poster session style, so don’t be hurt if not many stop and talk. Also, for your poster session (and for your oral presentation), it is a great idea to have a handout that summarizes your research. There are all sorts of aids and guides available that can advise you on how best to prepare your poster, including instructions in Beins and Beins (2008) and Landrum (2008). Per – sonally, I like to use Publisher to create single-sheet (large) posters, but many researchers who create the single-sheet posters use PowerPoint, and Beins and Beins (2008) provided excellent step-by-step instructions for this process. You may also want to consult Nicol and Pexman (2010) for additional advice about how to display your findings. 8.2 The Benefits of Undergraduate Research P sychology educators emphasize the skills gained from research courses because this empirical approach resides at the core of our discipline. There are clear ben – efits to undergraduate research participation, and not just for psychology majors. Broad research efforts have demonstrated these benefits nicely, and other work focuses more specifically on the benefits for psychology majors. The more undergraduate research experience you can garner, the better. These opportunities allow you to apply classroom knowledge as well as hone skills and abilities that are highly valued in the lan66845_08_c08_p229-258.indd 236 4/20/12 2:51 PM CHAPTER 8 Section 8.2 The Benefits of Undergraduate Research workplace. Plus, there is the added benefit of a mentoring relationship with a faculty member, which often results in good references and/or let – ters of recommendation. There are many good reasons for all of this empha – sis on research. First, it’s what psychologists do, and even if you do not go to graduate school, you will always be a consumer of research. So learning about how to be a savvy consumer of research is a valuable skill to possess. If faculty members are not providing research opportunities at your insti – tution, ask them to consider it (not all psychology faculty members are required to do research, but most work to maintain a scholarly knowledge of their subject area). You might even offer to share with the faculty member these helpful resources— Wayment & Dickson, 2008; Whiteside, Pantelone, Hunter-Reel, Eland, Kleiber, & Larimer, 2007—to help them frame the experience for you. Why would you want to go to the effort of orga – nizing and participating in an experience such as serving as a research assistant? In a study of stu – dents from multiple disciplines, Seymour, Hunter, Laursen, and Deantoni (2004) found that students self-reported gains in multiple areas, such as: • increase in confidence in ability to do research, increase in feeling like a scientist, increase in confidence in presenting/defending research, and establishing a men – toring/collegial relationship with faculty. • increase in critical thinking and problem-solving skills related to research, and increase in knowledge and understanding of science and research work. • increase in communication skills and increase in lab/field skills, measurement, work organization, and computer skills. • increase in longer-term interests in research and increase in commitment to aca – demic work. • increase in perception of undergraduate research because of its ability to provide real-world work experience; increase in opportunity to network with faculty, peers, and other scientists; enhancement of resume for graduate school applica – tion process. • increase in willingness to take on responsibility for a research project and increase in intrinsic motivation toward learning and attention to detail. Faculty members in psychology echo these benefits when asked about the benefits of par – ticipating in a research assistantship. Landrum and Nelsen (2002) reported faculty percep – tions about important aspects of research assistant experiences, and the faculty reported two major gains for students: technical skills are enhanced and interpersonal benefits are obtained. For example, the faculty reported that students acquired the technical skills to Undergraduate research can be beneficial for all students, not just psychology majors. What are some skills gained during your psychology education that you can apply outside of the classroom? Associated Press lan66845_08_c08_p229-258.indd 237 4/20/12 2:51 PM CHAPTER 8 Section 8.3 Key Organizations analyze data, prepare conference presentations, practice at manuscript preparation, and so on. The interpersonal benefits the faculty reported that students can reap from undergrad – uate research include networking with other students, making connections with faculty members, improving teamwork skills, and the opportunity to apply ethical principles to actual research situations. Not only can the undergraduate research opportunity be valu – able for those planning on applying to graduate school, but Sleigh and Ritzer (2007) made the case that research experience is also important preparation for the job market. In fact, participating in an undergraduate research experience might be more important for those students not going on to graduate school. It is a great opportunity to acquire skills that are beneficial to employers—in graduate school, graduate faculty members will ensure that research skills are attained and practiced, but research skills may not always be the first priority of future employers. What might you be asked to do if you serve as an undergrad – uate research assistant? Although described elsewhere (Landrum, 2008), Silvia et al. (2009) provide a good overview of the types of tasks. As you can see, there are many resources available to you to help you navigate the ins and outs of your undergraduate career. 8.3 Key Organizations I n the United States, there are three key national organizations that are perhaps most relevant to undergraduate psychology majors: The American Psychological Associa – tion (APA), the Association for Psychologi – cal Science (APS), and Psi Chi, the International Honor Society in Psychology. APA is the oldest of the three organizations, founded in 1892. Its first president was G. Stanley Hall, a person known in psychology for his series of “firsts” (includ – ing the first Ph.D. in psychology ever earned in the United States.). APA is the largest association of psychologists in the world, with over 150,000 members (APA, 2008). The mission of the Ameri – can Psychological Association is to “advance the creation, communication, and application of psychological knowledge to benefit society and improve people’s lives” (APA, 2008, para. 2). As an undergraduate student, you can become a stu – dent affiliate of APA, and if you are serious about psychology, I recommend it (and so do Silvia et al., 2009). As a student affiliate, you receive the American Psychologist , which is the APA flagship journal, and the APA Monitor on Psychology , which is a monthly magazine. You also have access to a website dedicated to student information (but you don’t need to be a student affiliate to access that information). Student affiliates also receive discounts on APA books, videos, other journal subscriptions, and online database products. G. Stanley Hall founded the American Psychological Association in 1892. Today, it is the oldest of the key psychological associations and has more than 150,000 members. Corbis/AP Images lan66845_08_c08_p229-258.indd 238 4/20/12 2:51 PM CHAPTER 8 Section 8.3 Key Organizations The other national organization for psychologists in the United States is the Association for Psychological Science, which was founded in 1988 as the American Psychological Society. The history of APS is interesting, because it was founded by many members who were previously members of APA. Today, APS has over 18,500 members world – wide, and they specialize in scientific, applied, and teaching aspects of psychology (APS, n.d.). APS has an undergraduate student affiliate program called the APS Student Caucus (APSSC), although it is primarily geared toward graduate students in psychol – ogy. APSSC also offers a twice-yearly publication called the Undergraduate Update . Simi – lar to APA, APS hosts an annual convention each year that undergraduate students can attend and participate in. I recommend that undergraduate psychology majors get in contact with their local faculty members to explore how involved they are in either of these organizations, and if you see an opportunity to attend a convention, or you real – ize the benefits of student affiliate membership, consider joining. Psychology, like every other discipline, is still an entity that relies on networking, and these organizations pro – vide excellent opportunities to network and become more connected in the (sometimes) small world of psychology. There is one international organization in psychology that is specifically focused on undergraduates studying psychology—Psi Chi. As an honor society, not every psychol – ogy major or minor is eligible to join. The membership requirements include having a GPA in the top 35% of your respective class (sophomores, juniors, or seniors), a minimum of 9 psychology credits completed at the time of application, sophomore standing, and having selected a psychology major or minor. There is a one-time application fee to the Psi Chi Central Office of $45 (at the time of this writing; a local chapter may add to the Central Office fee), and once you are a member, you are a member for life. The benefits of Psi Chi include documentation of your achievement (membership certificate, card, and pin), the opportunity to build your résumé as an undergraduate, and further opportuni – ties for professional growth (such as serving in a leadership capacity) (Psi Chi, 2009). But the benefits of Psi Chi are perhaps much deeper than those concrete benefits men – tioned above. Psi Chi can provide a sense of community among psychology majors, and this is sometimes hard to achieve at larger educational institutions. Not only can Psi Chi provide wonderful support for activities at a local level, but it is extremely active on a regional and national basis. For example, at each of the regional psychology association meetings mentioned in Table 8.2, Psi Chi organizes an extensive slate of programming just for students (regardless of Psi Chi membership). Each of those regions has a Psi Chi regional vice president who provides this programming and these opportunities for stu – dents at regional conventions and national conventions too, such as the APA convention typically held each August. Oftentimes the programming provided specifically focuses on leadership development, which is so valuable to undergraduate students regardless of their intended career path after graduation. If you explore the membership benefits of Psi Chi a bit further, you will discover that Psi Chi is generous at giving back to its members and faculty advisors. For the 2009 calendar year, Psi Chi allocated over $300,000 in awards and grants to student members, faculty members, and chapters (Psi Chi, 2009). If you join Psi Chi, plan on getting involved and being active. Yes, some people join for a line on the resume, but Psi Chi has the potential to be much more—like most opportunities in life, you get out of it what you put into it. lan66845_08_c08_p229-258.indd 239 4/20/12 2:51 PM CHAPTER 8 Section 8.4 Pursuing Graduate Work in Psychology 8.4 Pursuing Graduate Work in Psychology G iven the scope of this textbook, we can only go into a cursory overview of graduate school options in psychology and strategies for applying to graduate school. There are many good resources available, including: American Psychological Association. (2007). Getting in: A step-by-step plan for gain- ing admission to graduate school in psychology (2nd ed.). Washington, DC: Author. American Psychological Association. (2010). Graduate study in psychology . Wash – ington, DC: Author. Karcen, A. C., & Wallace, I. J. (Eds.). (2008). Applying to graduate school in psychol- ogy: Advice from successful students and prominent psychologists . Washington, DC: American Psychological Association. Keith-Spiegel, P., & Wiederman, M. W. (2000). The complete guide to graduate school admission: Psychology, counseling, and related professions (2nd ed.). Mahwah, NJ: Erlbaum. Wegenek, A. R., & Buskist, W. (2010). The insider’s guide to the psychology major: Everything you need to know about the degree and profession . Washington, DC: Amer – ican Psychological Association. Sayette, M. A., Mayne, T. J., & Norcross, J. C. (2010). Insider’s guide to graduate programs in clinical and counseling psychology (2010/2011 edition). New York, NY: Guilford Press. Walfish, S., & Hess, A. K. (Eds.). (2001). Succeeding in graduate school: The career guide for psychology students . Mahwah, NJ: Erlbaum. Depending on the specialty area in psychology you want to pursue, and the type of degree you wish to earn, graduate school can be extremely competitive or not so much. Nor – cross, Kohout, and Wicherski (2006) reported that approximately 27% of undergraduate psychology majors continue their education within two years of receiving their bach – elor ’s degree. Roughly speaking, more than 40,000 full-time students are enrolled in psychology gradu- ate programs. Table 8.3 provides a glimpse at the different specialty fields, with number of programs, average percentage of applicants accepted, and number of students enrolled in doctoral program and master ’s degree programs (from Norcross et al., 2006). Within two years of receiving their bachelor’s degrees in psychology, 27% of graduates choose to continue on with their education. What are your plans after you receive your degree? Digital Vision/Thinkstock lan66845_08_c08_p229-258.indd 240 4/20/12 2:51 PM CHAPTER 8 Section 8.4 Pursuing Graduate Work in Psychology Table 8.3: Graduate school admissions in departments of psychology by subfields Doctoral ProgramsMaster’s Degree Programs Subfield Number of programs Average percentage accepted Total students enrolled Number of programs Average percentage accepted Total students enrolled Clinical 21121.23,324 9852.71,671 Clinical Neuropsychology 20 25.8 213 Community 1231.0 432253.8416 Counseling 3421.5 447 108 65.52,764 Health 1230.9 87 370.3 23 School 5237.4 392 4948.9682 Other health services provider subfields 48 25.7 477 6464.51,395 Cognitive 8832.4 353 1052.8 25 Developmental 9927.2 374 1947.9166 Educational 3150.0 170 1557.3149 Environmental 239.1 10 Experimental 3137.6 163 3855.4261 Industrial/ organizational 53 25.7 281 7656.6849 General 5958.0972 Neuroscience 4926.9 148 632.3 50 Personality 1519.2 45 Physiological 439.4 12 Psychobiology 1325.0 34 Quantitative 1442.6 32 572.7 18 Social 8019.4 270 847.6 29 Other research subfields 76 33.2 339 4160.7443 Other fields 822.9 36 233.3 12 Total 98127.47,247 62457.49,925 Source: Norcross et al., 2006 lan66845_08_c08_p229-258.indd 241 4/20/12 2:51 PM CHAPTER 8 Section 8.4 Pursuing Graduate Work in Psychology The data in Table 8.3 provide some indication about the relative popularity of programs (number of students enrolled), as well as the competitiveness (average percentage accepted). One fact about Table 8.3 to keep in mind is that the category “doctoral pro – grams” includes both Ph.D. and Psy.D. programs across specialty fields. When someone applies to graduate school, it is typically a more involved process than applying to an undergraduate institution. Here are the types of information that you might be asked to provide in the graduate admissions process (from Landrum & Davis, 2010): (a) curricu – lum vitae or resume; (b) biographical statement or personal statements with your career interests and goals; (c) overall GPA, GPA in psychology, last two years GPA (verified with official transcripts); (d) list of relevant courses completed in the major; (e) Graduate Record Examination (GRE) scores (may include GRE Psychology Subject Test ); (f) let – ters of recommendation sent by you or sent directly from the school from (typically) three recommenders; and (g) application fee (if applicable). How do graduate admissions committees evaluate and weigh the different components of the graduate admissions package? The answer is complicated and varies greatly by school and type of degree program. But you can get a sense of what is important to master ’s degree and doctoral programs based on information from Landrum and Clark (2005) in Table 8.4 here. Table 8.4: “High importance” ratings for postgraduate degree program admission Doctoral Programs Master’s Degree Programs Admissions criterion Percent rated high importance Admissions criterion Percent rated high importance Letters of recommendation 86.7%Letters of recommendation 72.8% Statement of goals and objectives 83.3% G PA 68.7% G PA 70.9%Statement of goals and objectives 63.7% Research experience 69.2%Interview 47.0% Interview 63.1%GRE/MAT scores 39.3% GRE/MAT scores 53.1%Research experience 30.6% Clinically related public service 16.4%Clinically related public service 20.4% Work experience 15.1%Work experience 19.9% Extracurricular activity 3.8%Extracurricular activity 3.0% Source: Landrum and Clark (2005) One last piece of advice to offer before I point you in the direction of more resources for more in-depth answers: In studying mistakes students make in applying to gradu – ate school, Appleby and Appleby (2006) chronicled the “kisses of death” in the graduate school applications—avoid these mistakes in your own graduate admissions journey. The following presents the kisses of death to avoid in graduate school admissions. lan66845_08_c08_p229-258.indd 242 4/20/12 2:51 PM CHAPTER 8 Section 8.4 Pursuing Graduate Work in Psychology What to Avoid in the Graduate Admissions Process Personal Statements • Avoid references to your mental health. Such statements could create the impression that you may be unable to function as a successful graduate student. • Avoid making excessively altruistic statements. Graduate faculty members could interpret these statements to mean you believe a strong need to help others is more important to your success in graduate school than a desire to perform research and engage in other academic and profes- sional activities. • Avoid providing excessively self-revealing information. Faculty members may interpret such infor – mation as a sign that you are unaware of the value of interpersonal or professional boundaries in sensitive areas. • Avoid inappropriate humor, attempts to appear cute or clever, and references to God or religious issues when these issues are unrelated to the program to which you are applying. Admissions committee members may interpret this type of information to mean you lack awareness of the formal nature of the application process or the culture of graduate school. Letters of Recommendation • Avoid letters of recommendation from people who do not know you well, whose portrayals of your characteristics may not objective (e.g., a relative), or who are unable to base their descrip – tions in an academic context (e.g., your minister). Letters from these authors can give the impres – sion you are unable or unwilling to solicit letters from individuals whose depictions are accurate, objective, or professionally relevant. • Avoid letter of recommendation authors who will provide unflattering descriptions of your per – sonal or academic characteristics. These descriptions provide a clear warning that you are not suited for graduate study. Choose your letter of recommendation authors carefully. • Do not simply ask potential authors if they are willing to write you a letter of recommendation; ask them if they are able to write you a strong letter of recommendation. This question will allow them to decline your request diplomatically if they believe their letter may be more harmful than helpful. Lack of Information About the Program • Avoid statements that reflect a generic approach to the application process or an unfamiliarity with the program to which you are applying. These statements signal you have not made an honest effort to learn about the program from which you are saying you want to earn your graduate degree. • Avoid statements that indicate you and the target program are a perfect fit if these statements are not corroborated by specific evidence that supports your assertion (e.g., your research interests are similar to those of the program’s faculty). Graduate faculty members can interpret a lack of this evidence as a sign that you and the program to which you are applying are not a good match. Poor Writing Skills • Avoid any type of spelling or grammatical errors in your application. These errors are an unmis – takable warning of substandard writing skills, a refusal to proofread your work, or willingness to submit carelessly written work. • Avoid writing in an unclear, disorganized, or unconvincing manner that does not provide your readers with a coherent picture of your research, educational, and professional goals. A crucial part of your graduate training will be writing; do not communicate your inability to write to those you hope will be evaluating your writing in the future. Misfired Attempts to Impress • Avoid attempts to impress the members of a graduate admissions committee with information they may interpret as insincere flattery (e.g., referring to the target program in an excessively complimentary manner) or inappropriate (e.g., name dropping or blaming others for poor aca – demic performance). Graduate admissions committees are composed of intelligent people; do not use your application to insult their intelligence. Source: Appleby and Appleby (2006) lan66845_08_c08_p229-258.indd 243 4/20/12 2:51 PM CHAPTER 8 Section 8.5 Finding a Job with Your Psychology Degree Graduate school will resemble (in some respects) your undergraduate educational expe – rience, but Cox, Cullen, Buskist, and Benassi (2010) highlighted some critical transition issues from an undergraduate experience to a graduate education experience, including (a) no longer being the smartest person in your courses; (b) extremely high academic stan – dards; (c) increased reading of dense material; (d) balance of coursework with research, teaching, and working on a thesis or dissertation; (e) increased interactions with faculty members and peers; (f) increased emphasis on honing public speaking skills; (g) the need for exceptional time management skills; and (h) more mentoring with key faculty mem – bers to promote professional development. Looking back on your undergraduate career, some of these types of activities may have already occurred, but in graduate school, the stakes and expectations are higher. Making the most of your undergraduate experience and effectively networking with undergraduate peers and psychology faculty members can help ease the transition from your undergraduate educational experiences to your pending graduate education adventure. 8.5 Finding a Job with Your Psychology Degree I n psychology, the vast majority of baccalaureates (those who receive a bachelor ’s degree) will not continue on to graduate school in psychology. Even if you do attend graduate school, your eventual goal is a good job anyway, so the ability to secure a good job, regardless of the degree earned, will be helpful information at some point in your future. You should know that the career information about psychology majors that is available is not universally positive (e.g., Light, 2010; Rajecki & Borden, 2009), but that transitional materials are available (Briihl, Stanny, Jarvis, Darcy, & Belter, 2008; Hettich, 2010) to help make the most of your commencement toward your future. One of the most important discoveries that you can make about your future career is deter – mining, in essence, “What you want to be when you grow up?” Beware, because this is a much more complicated question than you might expect. Although it is extremely helpful to know what you don’t want to do, do you have an idea about what you would like to do for a living? Or do you know someone who has your perfect job? Self-exploration of the working world can be helpful in planning your career path (more on this toward the end of this chapter). It could be that you’ve had quite a bit of experience in the world of work, and you’ve come back to school because you realized you wanted to do something else. A great trait of colleges and universities is that they will be there whenever you are ready—education is truly a lifelong process and investment, and if you want to make a major career shift later in life the educational opportunities will be available. In fact, the career maturity theory (Crites, 1978) suggested that competent career choices can occur following accurate self-appraisal, the collection of occupational information, goal selec – tion, and future planning, all accompanied by proficient problem-solving abilities. If you are unsure about what your next steps are regarding career exploration, fulfilling each of these steps can provide you a solid foundation to build on. Exploring Career Options If you can, try to explore career opportunities and options during your undergraduate years. Visit with the experts at your campus career center and let them help you explore lan66845_08_c08_p229-258.indd 244 4/20/12 2:51 PM CHAPTER 8 Section 8.5 Finding a Job with Your Psychology Degree your interests and your voca – tional talents. Consider doing an internship so that you can see what others do for a living, day in and day out, with the educa – tion they have obtained. You might discover that to do what you want to do, more education is warranted, and sometimes more education in psychology might not be the best route toward your goal. Internships can be great because they pro- vide hands-on, real-world expe – rience. You may think that you’d like to work with developmen – tally disabled children for your career (a very noble goal). Com – pleting an internship with a local agency might confirm your beliefs—you may discover your “heart’s delight,” and know what you want to do. However, you might also realize that as much as you want to help, you don’t have the patience or think you can develop the skill set required to work with this population. As mentioned earlier, discovering what you don’t want to do is also valuable, and better to find this out sooner rather than later. Try finding that person who has your “perfect job” and inquire about his or her education and how he or she landed in that career. Also, in addition to the resources listed at the end of this section, take advantage of the power of the Internet in helping you find valuable career information. Try using O*NET , which is the Occupational Information Network (http://online.onetcenter.org ). O*NET presents details about over 1,000 different occupa – tions—including the knowledge skills needed, abilities needed, interests, general work activities, work contexts, and general salary information (Landrum & Davis, 2010). When Landrum and Harrold (2003) provided employers with a list of 88 potential skills and abilities and asked them to rate the importance of each one, employers reported the following top 10 desired skills and abilities. For you to be strategic during your career preparation, think about how you can acquire these skills and abilities across your under – graduate career. These skills take time to develop. Here’s the top 10 list, with 1 being the most important: 1. Listening skills 2. Ability to work with others as part of a team 3. Getting along with others 4. Desire and ability to learn 5. Willingness to learn new, important skills 6. Focus on customers/clients 7. Interpersonal relationship skills 8. Adaptability to changing situations 9. Ability to suggest solutions to problems 10. Problem-solving skills Undergraduates looking for ways to explore career options in their field should visit their school’s career center or ask about academic internship opportunities. Associated Press lan66845_08_c08_p229-258.indd 245 4/20/12 2:51 PM CHAPTER 8 Section 8.5 Finding a Job with Your Psychology Degree Once you have settled on some general ideas about your desired world of work, you can turn your attention to tasks that are relatively similar across many different job applica – tion situations. For instance, you are going to need to create a résumé and secure refer – ences for your job applications, you may be asked to interview, and you may also be asked for a work sample, if appropriate (by the way, if you are writing a formal paper for your course, your paper—revised, of course—could be used as a work sample). Not only are there books dedicated to each of the above tasks in the job search process, but some books cover all of these topics more in depth than we can go into here. The following tips pro – vide at least a glimpse at the type of information you will probably need to be successful in your job search. These tips (from Landrum & Davis, 2010), are some general résumé preparation tips for you to think about as you create or update your résumé. • Make the first impression count. A good résumé may get you to the next stage of the process. A poor résumé may stop you from going anywhere. • Keep your résumé current. Make sure it has your new phone number, email address, etc. • Make sure others proofread your résumé before you show it to potential employ – ers. Typographical and grammatical errors are unacceptable. Mistakes in your résumé will cost you the opportunity to advance in the employment process. • Have your résumé reviewed and critiqued by a career counselor and your men – tor in psychology. • Run a spell check and grammar check on your computer before showing your résumé to anyone. • Find a competent friend (an English major would be handy here) to do a gram – mar review of your résumé. • Then ask another friend to proofread it. The more sets of eyes that examine your résumé, the better. • Be concise—try to limit yourself to one to two pages. If the employer sets a page limit, follow it exactly. • Use white or off-white paper. • Use standard size, 8.5-inch × 11-inch paper. • Print on one side of the paper, using a font size between 10 and 14 points. • Use a non-decorative font (like Arial or Times New Roman); choose one font and stick to it. • Avoid italics, script, and underlined words. • Don’t use horizontal or vertical lines or shading. • Don’t fold or staple your résumé; if you must mail it, mail it in a large envelope. • Electronic résumés have different formatting demands. Many websites can help you prepare a web-friendly résumé. It is your responsibility to make sure that the company’s equipment is compatible when sending an electronic resume. At some point in the job application process, you will need to provide references for a potential employer to contact, or you may be asked to provide letters of recommendation. In any case, it would be wise to get to know some of the faculty members well enough that you can ask them to serve as a reference and/or write a letter of recommendation. As a tip, when you approach a faculty member about this topic, ask, “Would you be willing to write me a strong letter of recommendation?” If the faculty member asks you to remind him or her of your name, this is not a good sign. You are going to need to get to know your faculty lan66845_08_c08_p229-258.indd 246 4/20/12 2:51 PM CHAPTER 8 Section 8.5 Finding a Job with Your Psychology Degree members well enough so that they can speak to your professional skills and abilities. If you are at a large university, or you take large classes, you are probably not going to get to know your faculty mem – bers all that well, which means that it will take outside-of-class activities for them to get to know you. Those activities could mean participating as a research assis- tant or teaching assistant, serving as an intern, becoming involved in departmental activities, serv – ing as an officer in your local Psi Chi chapter, and a myriad of other ways of becoming involved. Get – ting to know faculty members also means that it will take time. The following list (from Landrum & Davis, 2010) presents additional, specific characteris – tics that can make for strong letters of recommendation. These are the characteristics that faculty members want to write about, so consider them: Do you possess these characteris – tics, and do you exhibit them on a regular basis to your psychology faculty? • Adapt to organizational rules and procedures • Comprehend and retain key points from written materials • Deal effectively with a variety of people • Display appropriate interpersonal skills • Exhibit effective time management • Gather and organize information from multiple sources • Handle conflict successfully • Hold high ethical standards and expect the same of others • Identify and actualize personal potential • Listen carefully and accurately • Plan and carry out projects successfully • Remain open minded during controversies • Show initiative and persistence • Speak articulately and persuasively • Think logically and creatively • Work productively as a member of a team • Write clearly and precisely At some point in the future, you will be invited to an interview. Perhaps you have already had job interviews. Interviews are typically a sign that you are a serious can – didate for the job you are applying for. Make sure you prepare for the interview effec – tively and think about the types of questions you may be asked. For general tips about interviewing, as well as potential questions you may be asked during an interview, A good résumé can help you stand out from others applying for the same job and get you to the next level of your career. iStockphoto/Thinkstock lan66845_08_c08_p229-258.indd 247 4/20/12 2:51 PM CHAPTER 8 Section 8.5 Finding a Job with Your Psychology Degree see Landrum and Davis (2010). How you present yourself is vitally impor – tant in these situations; make sure to utilize campus resources such as your Career Center and faculty men – tors who can help you prepare for the interview process. As promised, there are many more resources available for you about each of these topics and other impor – tant aspects of the job search process not mentioned here. Planning now, during your undergraduate years, is a wise investment that may help you navigate the path to the type of career that you want and the type of future you want for you and your family. Resources for information about a career with a bachelor ’s degree in psychology include: Hettich, P. I., & Helkowski, C. (2005). Connect college to career . Belmont, CA: Thomson Wadsworth. Kuther, T. L., & Morgan, R. D. (2007). Careers in psychology: Opportunities in a changing world (2nd ed). Belmont, CA: Thomson Higher Education. Landrum, R. E. (2009). Finding jobs with a psychology bachelor’s degree . Washington, DC: American Psychological Association. Landrum, R. E., & Davis, S. F. (2010). The psychology major: Career options and strat- egies for success (4th ed.). Upper Saddle River, NJ: Pearson Prentice-Hall. Morgan, B. L., & Korschgen, A. J. (2009). Majoring in psych? Career options for psy- chology undergraduates (4th ed.). Needham Heights, MA: Allyn & Bacon. Schultheiss, D. E. P. (2008). Psychology as a major: Is it right for me and what can I do with my degree? Washington, DC: American Psychological Association. Silvia, P. J., Delaney, P. F., & Marcovitch, S. (2009). What psychology majors could (and should) be doing . Washington, DC: American Psychological Association. Wahlstrom, C., & Williams, B. K. (2004). College to career: Your road to personal suc- cess . Mason, OH: South-Western. Wegenek, A. R., & Buskist, W. (2010). The insider’s guide to the psychology major: Everything you need to know about the degree and profession . Washington, DC: Amer – ican Psychological Association. Transition Tips for Success Expert advice about transitioning from college to career for psychology graduates comes from Hettich (2010), who outlined four different aspects of this college to career transition: The interview is a crucial step in the career process. You always want to be prepared and professional. Stockphoto/Thinkstock lan66845_08_c08_p229-258.indd 248 4/20/12 2:51 PM CHAPTER 8 Section 8.6 What Do You Want? What Will You Do? youth to adult, college to workplace, skills employers seek (rather than skills professors seek), and characteristics that help get you hired and promoted (and not get you fired or demoted). As you can imagine, there is a wealth of interesting and valuable information available, and in this context we can only skim the surface. For more details, consult Con- nect College to Career by Hettich and Helkowski (2005). For instance, if you want to keep your job, these are the behaviors you will want to avoid in the workplace—starting with the most frequent reasons new college graduate hires are fired: lack of work ethic and commitment, unethical behavior, failure to follow instruc – tions, missing deadlines, inappropriate use of technology, and being late for work (Het – tich, 2010). Although these typically don’t get you fired, they may lead to disciplinary action—ineffective teamwork, lack of initiative, inability to speak effectively, and inabil – ity to write effectively. Not to belabor the point, but many of these negative behaviors are behaviors that your college faculty also attempted to shape, such as imposing a late penalty for work not handed in on time, or having you adhere to the instructions of an APA-format-required assignment because paying attention to detail is a valuable trait in the workplace. To end on a more upbeat note, there is also advice for those transitioning from college to the workplace regarding how to get noticed and promoted (Hettich, 2010). Not surpris – ingly, these are often opposite the behaviors that will get you demoted or fired. If you are willing to take initiative, self-manage properly, display positive personal attributes, demonstrate commitment, exhibit leadership, and possess the ability to show and tell in an effective manner, you likely have the right stuff to excel in the workplace. Demonstrat – ing these qualities as a undergraduate student probably served you well and helped you achieve high grades too. 8.6 What Do You Want? What Will You Do? S elf-reflection is a good thing, and something that many of us do not do enough of. An important focus of this chapter is to provide information that is critical for you to be a successful psychology graduate. You are encouraged to explore career interest tools as well as life development ideas. In particular, I focus on the Self-Directed Search (SDS), a career-planning tool developed by John L. Holland (1994). The SDS developed out of Holland’s theories of vocational choice (1958, 1959). According to Holland (1973), four working assumptions drive the theory: 1. In this culture, most persons can be categorized as one of six types: realistic, investigative, artistic, social, enterprising, or conventional. 2. There are six kinds of environments: realistic, investigative, artistic, social, enter – prising, and conventional. 3. People search for environments that will let them exercise their skills and abili – ties, express their attitudes and values, and tackle agreeable problems and roles. 4. A person’s behavior is determined by an interaction between his or her personal – ity and the characteristics of his or her environment. The basic notion of this theory is that people are happier and more successful in a job that matches their interests, values, and skills. Scoring of the SDS is linked to occupational lan66845_08_c08_p229-258.indd 249 4/20/12 2:51 PM CHAPTER 8 Section 8.6 What Do You Want? What Will You Do? codes and titles. Thus, by deter – mining your preferences for styles or types, the SDS gives you some indication of the jobs that you might like and that would make the most of your skills and interests. The funda – mental idea is that people and work environments can be clas – sified according to Holland’s six types; thus, if you know your own type and understand the types that are associated with particular careers, you can find a match. Holland’s SDS (1994) is a relatively straightforward inventory. There is an Inter – net version (http://www.self -directed-search.com / ), which, for $9.95 (at the time of this writ – ing), you can take on your computer and receive a personalized report with your results. Individuals answer questions about their aspirations, activities, competencies, occupa – tions, and other self-estimates. These scores yield a three-letter Summary Code that des – ignates the three personality types the individual most closely resembles. With this code, test-takers use the Occupations Finder to discover which occupations best match their personality types, interests, and skills. This comprehensive booklet lists over 1,300 occu – pational possibilities—more than any other career interest inventory. Although it is not possible for you to take the SDS here, the six personality types and examples of corre – sponding careers are presented in Table 8.5. Table 8.5: Types and occupations of the self-directed search Realistic Investigative Personality Type OccupationsPersonality Type Occupations • Have mechanical ability and athletic ability? • Like to work outdoors? • Like to work with machines and tools? • Genuine, humble, modest, natural, practical, realistic? • Aircraft controller • Electrician • Carpenter • Auto mechanic • Surveyor • Rancher • Have math and science abilities? • Like to explore and understand things and events? • Like to work alone and solve problems? • Analytical, curious, intellectual, rational? • Biologist • Geologist • Anthropologist • Chemist • Medical technologist • Physicist (continued) The Self-Directed Search can help you find a career that fits your personality and skills best. Associated Press lan66845_08_c08_p229-258.indd 250 4/20/12 2:51 PM CHAPTER 8 Section 8.6 What Do You Want? What Will You Do? Table 8.5: Types and occupations of the self-directed search (continued) ArtisticSocial Personality Type OccupationsPersonality Type Occupations • Have artistic skills and a good imagination? • Like reading, music, or art? • Enjoy creating original work? • Expressive, original, idealistic, independent, open? • Musician • Writer • Decorator • Composer • Stage director • Sculptor • Like to be around other people? • Like to cooperate with other people? • Like to help other people? • Friendly, understanding, cooperative, sociable, warm? • Teacher • Counselor • Speech therapist • Clergy member • Social worker • Clinical psychologist Enterprising Conventional Personality Type OccupationsPersonality Type Occupations • Have leadership and public speaking ability? • Like to influence other people? • Like to assume responsibility? • Ambitious, extroverted, adventurous, self-confident? • Manager • Salesperson • Business executive • Buyer • Promoter • Lawyer • Have clerical and math abilities? • Like to work indoors? • Like organizing things and meeting clear standards? • Efficient, practical, orderly, conscientious? • Banker • Financial analyst • Tax expert • Stenographer • Production editor • Cost estimator The SDS presents some interesting career options. Although you haven’t taken the SDS, you can look at the six different types and realize that perhaps one or two of them fit you very well. The idea here is to not be afraid of some self-exploration; it is important for you to figure out what you would like to do for a career. College is a great time for career exploration; if you put some work into it, you will enjoy the rewards. Your career can take many different paths and progress through different stages or models. For instance, Harr (1995; as cited in Wahlstrom & Williams, 2004) differentiated job, occu – pation, and career this way: Your job is defined by the specific job duties that you fulfill within your occupation. Your occupation is the specific form that your career might take at any given time. Your career is the overall path you will take through your work life. There are different depictions of how a career might progress. Driver (1988; as cited in Wahlstrom & Williams, 2004) describes some of these career progressions: The linear career looks like climbing the stairs, in that you are climbing in the organization’s hierarchy. Each job along the way imparts more responsibility and requires more skill. In the steady-state career, you discover that you are comfortable with a particular occupation and you stay put. A promo – tion might mean more responsibility and more job stress, and you want to avoid that. The lan66845_08_c08_p229-258.indd 251 4/20/12 2:51 PM CHAPTER 8 Section 8.7 A Final Note spiral career suggests that one job builds on the other, being upwardly mobile. You might have a number of jobs that are different yet build on one another. Journalizing is powerful because answering powerful questions yields powerful, clear answers. When you write in a journal regularly, you become the type of person who can define what he or she wants, has definite plans, and can articulate desires. Combs (2000) suggests the following journalizing questions: • What are the activities that you love and enjoy most today? • What would be your ideal work environment today? • How would your ideal workday go today? • How would you define success today? • What might be your purpose or destiny? • How do you want to be perceived by your friends? Coworkers? Parents? Sig – nificant other? • What magazine would you most like to be featured in for your tremendous accomplishments in 10 years? • What would you like to be the best in the world at? • Who are your heroes and what is it about them that you most want to be like? • What do you really think should be changed in the world? • What do you most want to be remem – bered for at the end of your life? • Whom do you envy and what is it about them that you envy? As you can see, these are powerful questions that should provoke thoughtful responses. College is a good time for career exploration, but it is also a good time for life exploration as well. 8.7 A Final Note P sychologist Charles Brewer of Furman University once stated that “the fundamen – tal goal of education, from which all others flow, is to teach students to think as scientists about behavior” (1993, p. 169). If you adopt this as one of your goals of majoring in psychology and, perhaps equally important, if your instructors adopt this goal, then you begin to understand the centrality and importance of research methods like those discussed in this book. If you want to comprehend why most psychologists think of themselves as scientists, and if you want to understand why there is so much emphasis on methodology and statistical techniques, it is because our fundamental goal as psychology educators is to help our students begin to think like psychologists do. Amsel (2009) wrote about the importance of instilling these behaviors in undergraduate students, including Writing in a journal regularly can help you figure out your goals for the future. What other benefits might journalizing have? Flirt/SuperStock lan66845_08_c08_p229-258.indd 252 4/20/12 2:51 PM CHAPTER 8 Section 8.7 A Final Note students enrolled in an introductory psychology course, and he presented important ideas about the pedagogy (teaching practices) that psychology educators can use to help instill these cognitive strategies. But if you are interested in grasping how psychologists think, try reading Stanovich’s (2010) How to Think Straight About Psychology . Chapter by chapter, Stanovich presented illuminating examples of the fundamental characteristics of psycho – logical and scientific thinking, which include: • Falsifiability, and how we test hypotheses (disproving versus proving) • Operational definitions in psychology with precise measurement when possible • Reliance of replication (and not anecdotal evidence) • Correlation versus causation in psychological studies • Experimental control for meaningful comparisons • Laboratory versus field experiments, considering generalizations • Converging evidence and incremental support of hypotheses • Comprehending the role of chance in drawing conclusions Stanovich (2010) distilled three core beliefs for how psychologists think about behavior: systematic empiricism, replication and peer review of knowledge, and the search for test – able theories. The rules and techniques that you have learned about via this textbook have provided the foundation for systematic empiricism—psychologists want the results of observations to reveal something about the underlying interrelationships of the variables being studied (Stanovich, 2010). By methodically testing hypotheses and rejecting the null or failing to reject the null, we build a database of knowledge about human behavior founded upon systematic empiricism. In your reading of journal articles, the existence of the articles themselves reveals the frui – tion of replication and peer review. In fact, one of the reasons for the details presented in the Method section of journal articles is to provide enough information that a particular study could be replicated or repeated by someone else. After the process of writing a manuscript in APA style and format is complete, another process—peer review—must occur to enter the manuscript into the published literature. That is, experts within the field closely review all aspects of a work prior to the work’s publication in a scholarly journal. Publicly reporting the methodology used and openly inviting learned scholars to scruti – nize work prior to its publication creates a scientific approach to knowledge generation that is a key component of what it means to think like a psychologist. If you wish, you can think of the peer review process as “consumer protection” (Stanovich, 2010)—knowledge becomes “scientific” through this rigorous system of checks and balances that relies on very public involvement. The third core belief that psychologists hold is that the only testable theories are the sub – ject matter of science; that is, we seek to answer questions that can be arrived at through scientific means. That is, we attempt to test theories by applying psychology such that the result could lead to the support of or refutation of the theory. Stanovich (2010) described the process in psychology as theory prediction test theory modification As scientists, we seek out solvable problems that have hypotheses that can be supported or refuted—this means that inherently interesting questions, such as “Are humans good lan66845_08_c08_p229-258.indd 253 4/20/12 2:51 PM CHAPTER 8 Chapter Summary or evil,” “What is the meaning of life,” and “Does God exist” are not within the realm of scientific answers because we are unable to design a study such that the outcomes could support or refute the supposition. Now, we can certainly study people’s atti- tudes and opinions about good/ evil, the meaning of life, and God’s existence, but a prepon – derance of evidence (i.e., major – ity opinion) for one belief or other does indicate a causal con- nection. For causal conclusions about behavior, we need to uti – lize the tools and techniques pre – sented throughout this textbook to be able to ascertain causality. However, the principles you have read about throughout this book (and hopefully, what you have studied and applied in this course) can help provide a framework as a way of thinking like a psychologist and a method of answering complex questions about human behavior. Perhaps the most useful advice is to seek out your passions in life, whether they be personal and/or professional. If you can discover which aspects of your life you are passionate about, working hard toward meaningful goals becomes self-motivating and highly rewarding. We invest our time and talent in those aspects of life that we are passionate about. Whether you focus on your loved ones or your education or your pro – fession or your neighborhood or our planet, I encourage you to explore the passionate pursuits of your life and then use the knowledge, skills, and abilities that you acquire throughout your undergraduate career to help yourself and others. If you are fortunate enough to have one of your passionate pursuits overlap with what you do for a living, then going to work will hardly seem like going to work. Meeting these challenges is up to you; now go do it. Chapter Summary P sychology is an active discipline, and to be successfully active within it, one must go beyond reading and thinking about psychology but go out and “do” psychol – ogy. Thus, it may be completing an applied project after it has been proposed in a course. Psychologists network and get to know one another not only to further the sci – ence of psychology but also to make interpersonal connections that can lead to personal and professional benefits. National organizations exist that can help foster and provide opportunities to do psychology, and participation in these types of events can provide invaluable insights into the next steps in one’s career, whether that be pursuing a good job with your bachelor ’s degree in psychology or pursuing a graduate education in psychol – ogy (or in some other discipline). Multiple resources are provided for post-baccalaureate The knowledge you gained during this course can help you answer complex questions in your personal and professional life by using a psychological approach. Polka Dot/Thinkstock lan66845_08_c08_p229-258.indd 254 4/20/12 2:51 PM CHAPTER 8 Concept Check routes as well as specific transition information to ease the process. Tips for self-reflection and career exploration are provided, and students are encouraged to embrace these ideas and exercises. Finally, the ability to think like a psychologist is presented in the realm of critical thinking and how a general, evidence-based approach in psychology can be useful to broad applications of these critical thinking skills in daily life. Concept Check 1. In the research versus teaching debate in psychology, the text states that A. neither is more important. B. there is no debate in the field. C. teaching is more important. D. research is more important. 2. Regional psychology associations A. discourage networking among members. B. do not allow undergraduate participation. C. number 10 across the United States. D. each hold annual conventions. 3. Which of the following is NOT a recommendation for attending conferences? A. Be outgoing. B. Wear professional footwear. C. Pace yourself. D. Attend student events. 4. Silvia, Delaney, and Marcovitch (2009) associate what with “binge thinking”? A. Academic conferences B. Publishing research C. Networking with colleagues D. Conducting research 5. What percentage of people who hold bachelor ’s degrees in psychology pursue graduate degrees within two years of graduating (Norcross, Kohout, & Wicher – ski, 2006)? A. 5% B. 52% C. 27% D. 83% Answers 1. A. Neither is more important. The answer can be found Section 8.1. 2. D. Each hold annual conventions. The answer can be found Section 8.1. 3. B. Wear professional footwear. The answer can be found Section 8.1. 4. A. Academic conferences. The answer can be found Section 8.1. 5. C. 27%. The answer can be found in Section 8.4. lan66845_08_c08_p229-258.indd 255 4/20/12 2:51 PM CHAPTER 8 Key Terms to Remember Questions for Critical Thinking 1. Practicing some of these self-reflection skills, think about your Applied Project course—is it what you expected? Did you learn from the course what you needed to learn and what your instructor wanted you to learn? Did you maximize the opportunity, or perhaps was this a course to “get out of the way”? 2. You are near the end of your journey as an undergraduate psychology major. Given the sights you have seen over the course of the journey, would you do it all again? If you were starting again from Day 1, would you still be a psychology major? Why or why not? 3. To complete our big-picture thinking here, think about your educational choices. Have you truly engaged in your own educational processes and outcomes, or has your education been a series of hurdles to jump or tasks to get out of the way? What benefits have you reaped by being a student? What potential drawbacks may you face for being a student? If you were asked by a friend, family member, or loved one about your undergraduate experience, how would you describe it? Would you recommend this major and this university to a prospective student? Why or why not? This type of self-reflection can truly be helpful in shaping thoughts and perceptions of your current and future experiences. Key Terms to Remember American Psychological Association The oldest of the most important psychological organizations for undergraduates. Founded in 1892, it is the largest association of psy – chologists in the world, with over 150,000 members. APA’s mission statement is to “advance the creation, communication, and application of psychological knowledge to benefit society and improve people’s lives.” APA is commonly affiliated with organiza – tions such as Psi Chi. See Psi Chi. Association for Psychological Science An organization founded in 1988 as the American Psychological Society that cur – rently has over 18,500 members worldwide who specialize in scientific, applied, and teaching aspects of psychology. APS hosts annual conventions where students can get involved in the psychology field. conferences A forum for researchers to present their research findings in a public environment among individuals involved in a certain area of interest. These forums can be national, international, or regional in nature and may host oral and poster presentations. Council on Undergraduate Research An organization that provides an oppor – tunity for undergraduates to produce and present their research findings in a multi – disciplinary setting. Graduate Record Examination A stan- dardized test used to determine eligibility for graduate study that includes verbal, quantitative, and written categories and is typically required for the graduate school application process. GRE Psychology Subject Test A standard- ized test that focuses on a specific subject, such as psychology, to determine mastery and eligibility for graduate study. Certain graduate schools require completion of this exam for the application process. journalizing The process of documenting experiences that enables an individual to define what he or she wants, to plan, and to articulate desires. lan66845_08_c08_p229-258.indd 256 4/20/12 2:51 PM CHAPTER 8 Web Resources journals A forum for researchers and academics to publish and present find – ings from research. Typically they are peer reviewed academic sources that serve as a vessel to share, critique, and challenge up-and-coming ideas and findings in a particular field of study. McNair Scholars Program An organi- zation that provides an opportunity for student researchers to present research findings in a multidisciplinary setting as well as provides scholarships for students with excellent academic achievement. O*NET The occupational information net – work that provides details about more than 1,000 different occupations, including the knowledge skills needed, abilities needed, interests, general work activities, work con – texts, and general salary information. oral paper A method of presenting research in a conference setting where the presenter gives an oral presentation and speaks to listeners about the key points in their research study and then typically allows questions regarding said research. poster sessions A method of presenting research in a conference setting where the presenter creates a poster to sum up to main points of his or her research study and converse with conference attendees in a less formal setting than an oral presenta – tion. See oral presentation. Psi Chi The international honor society in psychology created to foster excellence of scholarship and advancement that is specifically geared toward psychology stu – dents. Membership requirements include having a GPA in the top 35% of your respective class (sophomores, juniors, or seniors), a minimum of 9 psychology cred – its completed at the time of application, sophomore standing, and having selected a psychology major or minor. Self-Directed Search A career interest inventory developed out of Holland’s the – ories of vocational choice that determine a job most compatible with a test taker ’s interests and strengths. Web Resources This website provides details of occupations for those who are interested in investigating potential job opportunities. http://online.onetcenter.org This website is where the international honor society in psychology has all information relevant to the organization and keeps members up to date on upcoming events and relevant publications and news. http://www.psichi.org This website outlines career options within the field of psychology and certain tools that you want to have to maximize your degree. http://career-advice.monster.com/job-search/company-industry-research/career-options- psychology-degree/article.aspx This website serves as an online psychology career center and lists career tips, job list – ings, internship opportunities, and other useful sources. http://www.socialpsychology.org/career.htm lan66845_08_c08_p229-258.indd 257 4/20/12 2:51 PM lan66845_08_c08_p229-258.indd 258 4/20/12 2:51 PM Glossary ABA design A single-subject design experiment with a baseline phase, fol- lowed by an intervention, and then fol- lowed by return to baseline. ABAB design A single-subject design experiment where the first A is the base- line, followed by the B intervention, followed by withdrawal of the interven- tion (returning to baseline—A), and then followed by the re-administration of the intervention again (B). This repeated administration and withdrawal of the intervention (independent variable) allows for greater confidence that the B phase is causing a change from baseline. abstract A quick synopsis of the main points of a research paper. APA style limits the length of an abstract to 120 words. alternate forms A test where a researcher develops two different forms of a test that are designed to be parallel but do not meet the same criteria levels for parallel forms. American Psychological Association The oldest of the most important psychological organizations for undergraduates. Founded in 1892, it is the largest association of psy – chologists in the world, with over 150,000 members. APA’s mission statement is to “advance the creation, communication, and application of psychological knowledge to benefit society and improve people’s lives.” APA is commonly affiliated with organiza- tions such as Psi Chi. See Psi Chi. anecdotal evidence Evidence based on a personal story or experience that is generally not considered to be scien- tific or empirical support for a hypoth- esis, but could contribute to hypothesis development. anonymity The absence of a connection between a specific participant and the data that he or she provides. A phase A single-subject design where baseline data are collected concerning the behavior of interest. ANOVA ANalysis Of VAriance A statisti- cal procedure that allows for the detection of differences when there are three or more levels of an independent variable, or two or more independent variables. APA style The writing style utilized for social science research results as outlined by the American Psychological Asso- ciation. This writing attempts to com- municate objectivity, credibility, and an evidence-based approach. archival research A research methodology that involves analysis of data from existing records that were made in natural settings. artifact When the measurement process is distorted, biased, or corrupted in some fashion. lan66845_09_glo_p259-270.indd 259 4/20/12 2:51 PM GLOSSARY Association for Psychological Science An organization founded in 1988 as the American Psychological Society that cur – rently has over 18,500 members worldwide who specialize in scientific, applied, and teaching aspects of psychology. APS hosts annual conventions where students can get involved in the psychology field. author notes A portion of the research paper located on the first page outlining specific notes or affiliations that an author wishes to reveal. availability sampling When the individu- als selected to participate were conve- niently available to the research. B phase The introduction of an interven- tion in a single-subject design. baseline Data that are collected at the beginning of an experiment to determine a starting point in the data collection process. beneficence An ethical principle found in the Belmont Report that states the poten- tial harm that research participants may experience must be balanced by the poten- tial benefits of the research. between groups design A method of study design is that intended to measure differences between separate groups of participants in a study. Ex: freshman, sophomores, juniors, and seniors. blocking A process of data analysis that turns a potentially extraneous variable into an independent variable, which permits the examination of whether or not the vari- able interacts with the intended indepen- dent variable. carryover effect The idea that the effect of one level of the independent variable can persist to influence another level of the independent variable. case study A research methodology that focuses on a particular case of interest. This case may be a person, a group, or per – haps an organization. Case studies can uti- lize qualitative and quantitative methods. causal relationship A direct relationship where an event occurs as a consequence of a previous event taking place. causation phase The phase of program evaluation in which the program evaluator attempts to demonstrate and measure the impact of the program. cause and effect An analysis that attempts to examine the causes and results of actions or behaviors. ceiling effect When a test does not have the ability to identify performance accurately because of the lack of difficult test items. changing criterion design A study design in which different participants, settings, or behaviors change gradually over time. cluster sampling The sampling practice of “clustering” groups of a population instead of evaluating each individual person to gain information when it is impossible or impractical to compile an exhaustive list of members composing the target population. coefficient of equivalence The correla- tion coefficient that results from a parallel forms test. See parallel forms. coefficient of stability A correlation coef- ficient that results from testing and retest- ing a score over time. cognitive dissonance theory A theory developed by Festinger and Carlsmith that occurs when a person privately holds an opinion but is pressured publicly to argue against the privately held opinion, and a form of discomfort or dissonance occurs because of the conflict. lan66845_09_glo_p259-270.indd 260 4/20/12 2:51 PM cohort A group of people who have shared experiences over the same span of time. cohort study A study design in which new samples of individuals are followed over time. communicability Communication of scientific, psychological knowledge by the reporting of results in a consistent and predictable format. concurrent validity The assessment of how the score on a test or inventory is related to your current state of affairs. confederate An individual who is part of a research study who acts as a participant during the research project. conferences A forum for researchers to present their research findings in a public environment among individuals involved in a certain area of interest. These forums can be national, international, or regional in nature and may host oral and poster presentations. confidentiality The experimenter ’s prom- ise not to reveal the results from a par – ticular individual unless that individual explicitly allows the experimenter to do so. confound An event or occurrence that happens at the same time of your study that is not part of your designed study but can influence its outcome. construct validity When a test measures what it purports to measure. Also known as “umbrella validity.” content analysis A method of analysis that takes qualitative statements, writings, and other forms of language and quanti- fies those data. content validity The determination as to whether or not the composition of items that make up a test measure reflects the universe of ideas, behaviors, and attitudes that compose the behavior of interest. convenience samples The sampling practice often used in exploratory research where a quick and inexpensive method is used to gather data by gathering partici- pants who are conveniently available for the purposes of data collection. Council on Undergraduate Research An organization that provides an oppor – tunity for undergraduates to produce and present their research findings in a multi- disciplinary setting. counterbalancing A technique that is used to minimize potential carryover effects in an experiment. covary To establish temporal precedence in a cause-and-effect relationship, the effect must be evident upon presentation of the cause. If there is no presentation of the cause, then there should be no effect. See temporal precedence. coverage The issue of who has Internet access and who does not that provides a barrier to obtaining information through Internet surveys. coverage error An error regarding the methodology used including access to Internet, use of land lines, and other methodologies. criterion-related validity The assessment of how the measurement outcome, or score, relates to other types of scores. critical thinking A set of strategies designed to make an individual a better consumer of information through inquiry, interpretation, and problem solving. lan66845_09_glo_p259-270.indd 261 4/20/12 2:51 PM GLOSSARY cross-sectional survey design A study design where data collection occurs at a single point in time with the population of interest. data analysis The process of interpreting data through statistical analysis into mean- ingful and accurate conclusions. data cleaning A method of reviewing data to ensure that it has been handled and entered accurately. debriefing A process that occurs at the con – clusion of a study that informs participants of the actual events that have occurred dur – ing the study, especially if deception was involved. degrees of freedom A statistical term that refers to the number of scores that are free to vary. demand characteristics When experimen- tal participants try to figure out the nature of the research and “help” the researcher by giving into the perceived demands. demographics Variables used to identify the traits of a study population. dependent variable The variable that is measured. determinism The theory that all events are predictable and that if all the causes were known for an event, that event would be completely predictable. Also known as the “lawfulness of nature.” dichotomous scale A scale in which there are only two possible responses, i.e., yes/ no, male/female, true/false. doi A digital object identifier code that is now used on some resources being pub- lished into the literature. The doi code pro- vides a unique numerical identifier of the permanent location of the electronic file on the Internet, primarily journal articles. double-blind experimentation When neither the study participants nor the experimenter are aware of the conditions being administered during the course of an experiment in order to prevent bias. empirical evidence Evidence produced by science. ethics The outlined principles that are fol- lowed by a given group or organization to uphold a moral code of conduct. external validity The assessment of whether or not a causal relationship can be generalized to other research settings, samples, or times in the event that a causal relationship has been determined to exist between the independent and dependent variables. face validity The assessment of whether or not the person taking the test believes that the test is measuring what is purports to measure. fact The result of careful observation that offers a description of an event or behavior. factorial design The statistical experi- ment design in which more than one independent variable is being manipu- lated, controlled, or arranged. This enables the experimenter to understand interactions between multiple indepen- dent variables. falsificationism The concept that the goal of science should not be to confirm or prove theories but rather to falsify or disprove theories. fatigue effect When an earlier trial negatively influences later results due to fatigue, boredom, or inattention. See nega- tive progressive error. fecundity The generation of new ideas; fruitfulness. See heuristic value. lan66845_09_glo_p259-270.indd 262 4/20/12 2:51 PM GLOSSARY finite causation The concept that there are a limited number of causes for any effect or event, and that these causes are discov- erable and understandable. fixed-effect variable A variable assumed to be measured without error. floor effect Occurs when you are work- ing with scores at the very low end of the distribution of scores that do not have the potential to go any lower. Thus, sub- sequent attempts yielding improvement may not be accurate because of the scores’ inability to decrease. generation phase The phase of program development where a needs assessment may be conducted prior to the imple- mentation of a program, to see if the need justifies the implementation of a program, followed by program development and refinement. Graduate Record Examination A stan- dardized test used to determine eligibility for graduate study that includes verbal, quantitative, and written categories and is typically required for the graduate school application process. GRE Psychology Subject Test A standard- ized test that focuses on a specific subject, such as psychology, to determine mastery and eligibility for graduate study. Certain graduate schools require completion of this exam for the application process. Guttman scale A survey response scale that generates a set of items that increase in difficulty. If a participant agrees with one scale item, it is assumed that they agree with the preceding scale items. heuristic value A theory’s ability to motivate others to conduct research on the topic and generate new ideas about the world that we live in. See fecundity. homogeneous Variables or conditions that are similar in nature. hypothesis An educated guess that attempts to explain the facts that result from scientific studies. implementation phase The phase of program development where the program begins, provides services to clients and the community, and hopefully has a posi- tive impact. independent variable The variable that is manipulated, controlled, or arranged/ organized by the researcher. in-person interviews A research method- ology that allows an interviewer and a par – ticipant to build rapport through conversa- tion and eye contact, which might allow for deeper questions to be asked about the topic of interest. This presents fewer limitations about the types and length of survey items to be asked. Institutional Review Board (IRB) Institu- tions that approve and investigate research studies and protocols to ensure ethical consideration is given to the protection of human subjects. intentional plagiarism Purposeful and intentional cheating often due to procrasti- nation and panic about assignment dead- lines. Not giving credit for ideas and work that has been previously produced from other sources. interactions An effect that allows us to look at the combinations of the levels of the independent variables to examine if these combinations lead to different outcomes compared to other possible combinations. lan66845_09_glo_p259-270.indd 263 4/20/12 2:51 PM GLOSSARY internal validity The assessment of the general nature of the relationship between the independent variables and the depen- dent variables. It primarily focuses on the determination of causality and whether or not the manipulation of the independent variables caused changes in the dependent variables. interrater reliability A method of deter – mining reliability in which two or more raters categorize nominal data and obtain the same result when using the same instrument to measure a concept. intersubject variability The ability of a single-subject design to identify variation in different individuals. intersubjective testability When a theory generates hypotheses that are testable from an empirical standpoint. interval/ratio An interval scale presents numbers in a meaningful way and pro- vides equal intervals including zero. In a ratio scale, numbers are used in the typical where 0 = a lack of something. The two scales of measurement are usually com- bined in psychological research since their interpretation individually can present challenges. introduction The portion of a research paper that provides the reader with a context for everything that is going to be investigated. This includes introducing the research problem, developing the background, and stating the purpose and rationale for the research paper, including hypotheses. journalizing The process of documenting experiences that enables an individual to define what he or she wants, to plan, and to articulate desires. journals A forum for researchers and academics to publish and present find- ings from research. Typically they are peer reviewed academic sources that serve as a vessel to share, critique, and challenge up-and-coming ideas and findings in a particular field of study. justice An ethical principle found in the Belmont Report that the burden of research does not fall exclusively on any one group or class of individuals in society. law A generalization for a collection of facts, but without explanation. Scientific laws are identified when no exceptions have been found to the law; scientific laws explain what has happened. Likert scale A survey response scale that has a 5-point scale, measuring from one pole of disagreement to the other pole of agreement with each of the scale points having a specific verbal description. longitudinal survey A study design where data collection occurs at several points over an extended period of time. main effects The overall effect of each of the independent variables considered individually. matching The pairing of participants based on similar measures on a targeted variable. materials A portion of a research paper included in the method section that pro- vides the details of the actual items or objects that were used to carry out the study. McNair Scholars Program An organi- zation that provides an opportunity for student researchers to present research findings in a multidisciplinary setting as well as provides scholarships for students with excellent academic achievement. lan66845_09_glo_p259-270.indd 264 4/20/12 2:51 PM GLOSSARY measurement How the responses of indi- viduals are captured for the purposes of research. measurement error An error that can occur due to a number of reasons, typi- cally including measurement variation and measurement bias. meta-analysis A method of analysis that provides an approach for combining the results of multiple studies so that general effects can be summarized. method The section of a research paper that outlines how the study was conducted so that other researchers can replicate your study. The subsections include partici- pants, materials, and procedure. mixed design When an experimenter includes both between groups and within groups design features into his or her research. mixed-mode approach A study design where multiple research modalities are accessed to achieve the research goals. multiple baseline design The study design in which there are multiple base- lines for comparison. multistage sampling The two-stage sam- pling practice involving the formation of clusters as a primary selection, then sam- pling members from the selected clusters to produce a final sample. naturalistic observation An observational situation where the researcher does not interact in the environment, but merely observes it. negative progressive error When an ear – lier trial negatively influences later results due to fatigue, boredom, or inattention. See fatigue effect. nonequivalent control groups design A design in which an experimenter is unable to randomly assign participants to the different groups due to external factors. nonprobability sampling The sampling practice where the probability of each participant being selected for a study is unknown and sampling error cannot be estimated. See convenience sampling, quota sampling, snowball sampling, and volunteer sample. non-response error An error occurring when there is a response rate of 25% or less for a particular question. nonsense syllables A consonant-vowel- consonant combination that is pronounce- able but has no inherent meaning, created by Ebbinghaus for memory research on himself. non-subject variable When the value of the independent variable is not determined by the participant but rather by the researcher. O*NET The occupational information net- work that provides details about more than 1,000 different occupations, including the knowledge skills needed, abilities needed, interests, general work activities, work con – texts, and general salary information. operational definition A concise defini- tion that exhibits precisely what is being measured. oral paper A method of presenting research in a conference setting where the presenter gives an oral presentation and speaks to listeners about the key points in their research study and then typically allows questions regarding said research. panel study A study design in which the same people are studied over time, span- ning at least two points in time. lan66845_09_glo_p259-270.indd 265 4/20/12 2:51 PM GLOSSARY parallel forms A test where a researcher administers two versions of a test to the same group of individuals, resulting in a correlation of the outcomes between the two test administrations. See coefficient of equivalence. parsimony When a theory or idea is simple, yet complete. participants The portion of a research paper included in the method section that describes the characteristics of the indi- viduals who completed your study. pilot test A “practice run” of a ques- tionnaire used to determine weaknesses and optimize the length of the scale for adequate response rate. The conditions in which the survey will be administered are typically replicated as closely as possible to the actual survey administration. plagiarism When you borrow intellectual property without crediting the original source. See intentional plagiarism and unintentional plagiarism. planned comparison When an experi- menter decides which comparisons to conduct when the experiment is being designed and develops questions relevant to that comparison. plausible alternative explanation The ability to state, with confidence, that there is no other logical explanation for the effect other than the cause. positive progressive error When perfor – mance on an earlier trial in the experiment positively influences later results due to practice, experience, or familiarity. See practice effect. post hoc analyses Analyzing the data after the experiment has been conducted to find patterns that were not outlined in the experiment development. poster sessions A method of presenting research in a conference setting where the presenter creates a poster to sum up the main points of his or her research study and converse with conference attendees in a less formal setting than an oral presenta- tion. See oral presentation. posttest only design A study design that only analyzes the measures of a sample after an intervention is administered. practice effect When performance on an earlier trial in the experiment positively influences later results due to practice, experience, or familiarity. See positive pro- gressive error. predictive validity When a researcher takes current knowledge and attempts to make a prediction about the future. pretest-posttest design A study design that analyzes measures of a sample both before and after an intervention is administered. probability sampling The sampling prac- tice where the probability of each partici- pant being selected for a study is known and sampling error can be estimated. See simple random sampling, systematic sampling, stratified sampling, cluster sam- pling, and multistage sampling. procedure The portion of the research paper included in the method section that guides the reader through the process you used to conduct your research, step by step in chronological order. program evaluation An assessment of the impact of a program based on its pre- stated goals. progressive error When factors other than the independent variable are influencing the dependent variable over time. lan66845_09_glo_p259-270.indd 266 4/20/12 2:51 PM GLOSSARY Psi Chi The international honor society in psychology created to foster excellence of scholarship and advancement that is specifically geared toward psychology stu- dents. Membership requirements include having a GPA in the top 35% of your respective class (sophomores, juniors, or seniors), a minimum of 9 psychology cred- its completed at the time of application, sophomore standing, and having selected a psychology major or minor. psychology The science of human behavior. qualitative variable A variable in which responses differ in kind or type. The outcomes of these variables are usually described in words. quantitative variable A variable for which there is some known entity. The outcomes of these variables are usually described in numbers. quasi-experiment When a participant is not randomly assigned to a group but instead assigned to a group based on char – acteristics that he or she already possesses. quota sampling The sampling practice where a researcher identifies a target population of interest and then recruits individuals (non-randomly) of that popu- lation to participate in a study. random assignment When participants are randomly assigned to a group or condition in an attempt to control for any significant differences among groups. random-effect variable The condition where a sample is drawn from a popula- tion it hopes to represent, such as select- ing participants from a population into a sample. randomization When individuals are assigned to a study group by chance and not in a predictable manner. randomized block design When partici- pants are grouped into blocks based on a determined variable and then the blocks are randomly split for assignment to an experimental or control group. references The section of the research paper that contains a listing of every cita- tion that you used in the paper. regression discontinuity design A study design in which assignment into the treat- ment group or the control group is based on a predetermined cutoff. regression toward the mean When an experimenter sees a change in scores and thinks the independent variable is effec- tive, but this change in scores is due to something else. repeated measures design When the experimenter wishes to examine change over time by administering a condition to the same participant over a period of time. replication The ability to repeat a study. representative The assumption that a sample will resemble all qualities of the general population to ensure that results of a sample can be applied to the whole general population. representativeness A challenge in Internet surveys regarding whether or not results obtained from this method are representa- tive of the entire population. respect for persons An ethical principle found in the Belmont Report that led to the requirement of informed consent; that is, human participants deserve to know the risks involved in research and what their protections are. response set A pattern of responding seen in a participant that may not accurately reflect the participant’s true feelings on a topic. lan66845_09_glo_p259-270.indd 267 4/20/12 2:51 PM GLOSSARY response set acquiescence When participants get stuck in the trend of responding yes repeatedly in a survey or questionnaire. response set social desirability When participants respond in a pattern that they believe makes them look good, or look bet- ter than they are. results The section of the research paper that tells the reader the outcomes of the study, typically from a quantitative or qualitative viewpoint. This section pres- ents the data; it does not interpret them. reversal designs A study design involving the application and removal of an inter – vention strategy. roughly equivalent groups Obtaining groups that are as similar to one another as possible through randomization or another technique because of the unlikelihood of obtaining exactly equivalent groups. running head A heading in the research paper that appears at the top right-hand corner with the first five characters of the paper title in upper case letters. The words “Running head” only appear on the first page of the paper. sampling error An error occurring when all the potential participants from a popu- lation may not be represented in a sample. savings Established by Ebbinghaus, the difference in time between when something is first learned and when it is re-learned. scale A tool used to measure a person’s attitudes, perceptions, behaviors, etc. that is chosen to best represent a study. scales of measurement Tools used to translate observations into scores in nomi- nal, ordinal, interval, or ratio scales. scientific method A method of studying the world around us by observing and developing theories through scientific hypotheses. Self-Directed Search A career interest inventory developed out of Holland’s the- ories of vocational choice that determine a job most compatible with a test taker ’s interests and strengths. semantic differential scale A survey response scale used to measure affect and emotion using dichotomous pairs of words and phrases that a participant evaluates on a scale of 1 to 7. simple effects A statistical test that seeks to identify how the one condition is different from alternative combinations of other inde – pendent variables in the same experiment. simple random sample The purest form of sampling practice, and probably one of the rarest techniques used where every- body in the survey population has the same probability of being tested. single-blind experiment When the par – ticipant is unaware of the experimental condition he or she is in. single-subject design A study design in which cause-and-effect conclusions can be approximated with limited generalizability since the data are typically based on one participant, who serves as his or her own control. skepticism The potential doubt that others may feel regarding the findings of scientific analysis. Scientists value this because they want evidence to either support or refute a claim. snowball sample The sampling practice where members of the target population of interest are asked to recruit other members of the same population to participate in the study. lan66845_09_glo_p259-270.indd 268 4/20/12 2:52 PM GLOSSARY sphericity The correlation between the multiple scores in a repeated measures design. split-half method A method of estimating internal consistency that involves splitting the instrument in half and then correlating the scores from the resulting halves. split-plot design A type of mixed design, and a factorial design where experimen- tal conditions are grouped, such as study guide versus no study guide, and are then separately compared to different sub- groups, such as easy questions, medium questions, and hard questions, to accu- rately determine effectiveness. stable baseline When a participant’s behavior is consistent previous to par – ticipation in a study to ensure accurate measurement once an intervention is administered. statistical conclusion validity The assessment of whether or not method- ological and statistical approaches used in an experimental situation are sensitive enough to capture a causal relationship. statistical relationship A relationship between two variables that is found through the analysis of statistical measures. stratified sampling The practice of divid- ing a sample into subcategories (strata) in a way that identifies existing subgroups (such as gender) in a general population to make a sample the same proportion as displayed in a population. subject variable A characteristic, such as GPA, that an experimenter cannot randomly assign because the participant already has that characteristic before par – ticipating in the study. subsumptive power The ability of a the- ory to account for the results of prior stud- ies while offering a theoretical framework. systematic random sample The sampling practice in which every nth person from a sample is selected. temporal precedence To determine what is the cause and what is the effect, the cause must come first and the effect must come second. If they occur at the same point in time, then the determination of which is the cause and which is the effect cannot be made. theory An attempt to explain facts that are often tested as research hypotheses. Thurstone scale A survey response scale developed to measure attitude by creat- ing a response scale of equally appearing intervals by having participants make a series of comparative judgments. time series design A study design that includes consistent and multiple observa- tions of a variable that is repeated a sub- stantial number of times. unintentional plagiarism When citation guidelines are not strictly followed to give credit where credit is due. Typically occurs as a result of careless note-taking practices, misunderstanding of citation rules, citing uninformed opinions, or following APA rules for citation in a sloppy manner. validity The determination as to whether or not researchers are truly “measuring what they think they are measuring” for the purposes of their research. values of science Outlined values that state that science places high value on theories that have the largest explanatory power; science values fecundity; science values open-mindedness; scientists require logical thinking in their explanations; science values skepticism; and science is self-correcting. variable An entity that can take on differ – ent values. lan66845_09_glo_p259-270.indd 269 4/20/12 2:52 PM GLOSSARY visual analog scale A survey response scale used to obtain a score along a contin- uum, where a participant places a check- mark to indicate where his or her attitude or opinion falls along the scale. volunteer sample The common sampling practice where volunteers are asked to participate in a survey. volunteerism An error occurring when there is not enough self-selection in a study. within groups design An experiment design that aims to measure the change within a participant over time. lan66845_09_glo_p259-270.indd 270 4/20/12 2:52 PM GLOSSARY Adair, J. G. (1984). The Hawthorne effect: A reconsideration of the methodological arti- fact. Journal of Applied Psychology, 69, 334–345. Adler, P. A., & Adler, P. (1004). Observational techniques. In N. Denzin, & Y. Lincoln (Eds.), Handbook of qualitative research (pp. 377–392). Thousand Oaks, CA: Sage Publications. Agre, P., & Rapkin, B. (2003). Improving informed consent: A comparison of four consent tools. IRB: Ethics & Human Research, 25, 1–7. Aiken, L. R., & Groth-Marnat, G. (2006). Psychological testing and assessment (12th ed.). Boston, MA: Pearson/Allyn & Bacon. Altermatt, B. (2008). Threats to internal validity for within-subjects designs. Retrieved from htt p://vault.hanover.edu/~altermatt w/methods/assets/Readings/Within _subjects.pdf Alwin, D. F., McCammon, R. J., & Hofer, S. M. (2006). Studying baby boom cohorts within a demographic and developmental context: Conceptual and methodolo gi – cal issues. In S. K. Whilbourne, & S. L. Willis (Eds.), The baby boomers grow up: Contemporary perspectives on midlife (pp. 45–71). Mahwah, NJ: Erlbaum. American Psychological Association. (2007). Getting in: A step-by-step plan for gaining admission to graduate school in psychology (2nd ed.). Washington, DC: Author. American Psychological Association. (2008). About the American Psychological Association. Retrieved from htt p://www.apa.org/about/ American Psychological Association. (2010). Graduate study in psychology. Washington, DC: Author. American Psychological Association. (2010). Publication manual of the American Psychologi- cal Association (6th ed.). Washington, DC: Author. American Psychological Society. (n.d.). Association for Psychological Science: History of APS. Retrieved from http://www.psychologicalscience.org/about/history.cfm References lan66845_10_ref_p271-294.indd 271 4/20/12 2:51 PM Amsel, E. (2009). Promoting psychology students’ adoption of core disciplinary knowledge and values: Three pedagogical themes. Unpublished manuscript, Department of Psychol- ogy Weber State University. Angrosino, M. V. (2007). Naturalistic observation. Walnut Creek, CA: Left Coast Press. Appleby, D. (2005). Honors B104 Psychology as a Social Science. Retrieved from Project Syl- labus, Office of Teaching Resources in Psychology, http://teachpsych.org/otrp/syllabi/syllabi.php?category=Introductory%20 Psycholog y Appleby, D. C., & Appleby, K. M. (2006). Kisses of death in the graduate school applica- tion process. Teaching of Psychology, 33, 19–24. Aronson, E. (1992). The return of the repressed: Dissonance theory makes a comeback. Psychological Inquiry, 3, 303–311. Ator, N. A. (2005). Conducting behavioral research: Methodological and laboratory animal welfare issues. In C. K. Akins, S. Panicker, & C. L. Cunningham (Eds.), Laboratory animals in research and teaching: Ethics, care, and methods (pp. 43–68). Washington, DC: American Psychological Association. Attwood, R. (2009, March 26). I can’t hear you . . . Times Higher Education [Online edition]. Retrieved from http://timeshighereducation.co.uk/story.asp?storyCode=405877&sectioncode=26 Ayllon, T., Layman, D., & Kandel, H. J. (1975). A behavioral-educational alternative to drug control of hyperactive children. Journal of Applied Behavior Analysis, 8, 137–146. Babbie, E. R. (1973). Survey research methods. Belmont, CA: Wadsworth Publishing. Bangert-Drowns, R. L., & Rudner, L. M. (1991). Meta-analysis in educational research. Practical Assessment, Research & Evaluation [Online journal], 2. Retrieved from http://pareonline.net/getvn.asp?v=2&n=8 Baritz, L. (1960). Hawthorne. In The servants of power: A history of the use of social science in American industry (Chapter 5). Middletown, CT: Wesleyan University Press. Barlow, D., & Hersen, M. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). Elmsford, NY: Pergamon Press. Baumeister, R. F. (1995). Disputing the effects of championship pressures and home audi – ences. Journal of Personality and Social Psychology, 68, 644–648. Baumeister, R. F. (2008). Free will in scientific psychology. Perspectives on Psychological Science, 3, 14–19. Bawden, D. L., & Sonenstein, F. L. (n.d.). Quasi-experimental designs. Retrieved from http://www.childwelfare.com/courses/Documents/bawden.htm lan66845_10_ref_p271-294.indd 272 4/20/12 2:51 PM REFERENCES Beck, H. P., Levinson, S., & Irons, G. (2009). Finding Little Albert: A journey to John B. Watson’s infant laboratory. American Psychologist, 64, 605–614. Becker, L. A. (1999). Scales of measurement. Retrieved from http://web.uccs.edu/lbecker/SPSS/scalemeas.htm Beer, M., & Cannon, M. D. (2004). Promise and peril in implementing pay-for-perfor – mance. Human Resource Management, 43, 3–48. doi:10.1002/hrm.20001 Beidernikl, G., & Kerschbaumer, A. (2007). Sampling in online surveys. In R. A. Reyn – olds, R. Woods, & J. D. Baker (Eds.), Handbook of research on electronic surveys and measurements (pp. 90–96). Hershey, PA: Idea Group Reference/IGI Global. Beins, B. C., & Beins, A. M. (2008). Effective writing in psychology: Papers, posters, and pre- sentations. Malden, MA: Blackwell. Bem, D. J. (2004). Writing the empirical journal article. In J. M. Darley, M. P. Zanna, & H. L. Roediger, III (Eds.), The compleat academic: A career guide (2nd ed.) (pp. 185–219). Washington, DC: American Psychological Association. Benjamin, L. T., Jr., & Simpson, J. A. (2009). The power of the situation: The impact of Milgram’s obedience studies on personality and social psychology. American Psy- chologist, 64, 12-19. doi:10.1037/a0014077 Binik, Y. M, Mah, K., & Kiesler, S. (1999). Ethical issues in conducting sex research on the Internet. Journal of Sex Research, 36, 82–90. Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832. doi:10.1146/annurev.psych.55.090902.141601 Bishop, S. R., Lau, M., Shapiro, S., Carlson, L., Anderson, N. D., Carmody, J., Segal, Z. V., Abbey, S., Speca, M., Velting, D., & Devins, G. (2004). Mindfulness: A pro- posed operational definition. Clinical Psychology: Science and Practice, 11, 230–241. doi:10.1093/clipsy/bph077 Blackwell, J. (n.d.). 1935: The poll that took America’s pulse. The Trentonian. Retrieved from http://www.capitalcentury.com/1935.html Blinn-Pike, L., Worthy, S. L., & Jonkman, J. N. (2007). Disordered gambling among col – lege students: A meta-analytic synthesis. Journal of Gambling Studies, 23, 175–183. doi:10.1007/s10899-006-9036-2 Boring, E. G. (1950). A history of experimental psychology. Englewood Cliffs, NJ: Prentice-Hall. Braunsberger, K., Wybenga, H., & Gates, R. (2007). A comparison of reliability between telephone and web-based surveys. Journal of Business Research, 60, 758–764. doi:10.1016/j.busres.2007.02.015 lan66845_10_ref_p271-294.indd 273 4/20/12 2:51 PM REFERENCES Brewer, C. L. (1993). Curriculum. In T. V. McGovern (Ed.), Handbook for enhancing under- graduate education in psychology (pp. 161–182). Washington, DC: American Psycho- logical Association. Bridgman, P. W. (1927). The logic of modern physics. New York, NY: MacMillan. Briihl, D. S., Stanny, C. J., Jarvis, K. A., Darcy, M., & Belter, R. W. (2009). Thinking criti- cally about careers in psychology. In D. S. Dunn, J. S. Halonen, & R. A. Smith (Eds.), Teaching critical thinking in psychology: A handbook of best practices (pp. 225–234). Malden, MA: Wiley-Blackwell. Brody, J. L., Gluck, J. P., & Aragon, A. S. (2000). Participants’ understanding of the pro- cess of psychological research: Debriefing. Ethics & Behavior, 10, 13–25. Brown, G. G. (n.d.). Naturalistic observation and case-study research. Retrieved from http://polar.alaskapacific.edu/gregb/ES601/ch6notes.html Brown, R. T., & Borden, K. A. (1986). Hyperactivity at adolescence: Some misconceptions and new directions. Journal of Child Clinical Psychology, 15, 194–209. Bryan, A. J. (1987). Single-subject designs for evaluation of sport psycholog y interven – tions. Sport Psychologist, 1, 283–292. Buckley, K. W. (1989). Mechanical man: John Broadus Watson and the beginnings of behaviorism. New York, NY: Guilford Press. Burger, J. M. (2009). Replicating Milgram: Would people still obey today? American Psy- chologist, 64, 1-11. doi:10.1037/a0010932 Burns, P. A. (1997). Securing internal validity—how to avoid the threats. Retrieved from http://www.socialresearchmethods.net/tutorial/Burns/int.html Busch, C., De Maret, P. S., Flynn, T., Kellum, R., Le, S., Meyers, B., Saunders, M., White, R., & Palmquist, M. (2009). Content analysis (Colorado State University, Depart- ment of English). Retrieved from http://writing.colostate.edu/guides/research/content Camp, B. (1995). The problem of matching. Developmental and Behavioral Pediatrics, 16, 54–55. Campbell, J. M. (2004). Statistical comparison of four effect sizes for single-subject designs. Behavior Modification, 28, 234–246. doi:10.1177/0145445503259264 Cantrell, M. A., & Lupinacci, P. (2007). Methodological issues in online data collection. Journal of Advanced Nursing, 60, 544–549. doi:10.1111/j.13652648.2007.04448.x Cardinal, B. J. (2002). Decisional Balance Scale for exercise behavior: A brief commentary with guidelines for item construction. Measurement in Physical Education and Exer- cise Science, 6, 249–253. lan66845_10_ref_p271-294.indd 274 4/20/12 2:51 PM REFERENCES Carr, J. E., & Austin, J. (1997). A classroom demonstration of single-subject research designs. Teaching of Psychology, 24, 188–190. Carrier, R. (2001). Test your scientific literacy! Retrieved from http://www.infidels.org/library/modern/richard_carrier/SciLit.html Chancey, L., & Dumais, S. A. (2009). Voluntary childlessness in marriage and family textbooks, 1950–2000. Journal of Family History, 34, 206–223. doi:10.1177/ 0363199008330733 Chiang, H.-M. (2008). Communicative spontaneity of children with autism: A prelimi – nary analysis. Autism, 12, 9–21. doi:10.1177/1362361307085264 Cho, H., & LaRose, R. (1999). Privacy issues in Internet surveys. Social Science Computer Review, 17, 421–434. Chromy, J. (2006). Survey sampling. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 641–654). Washington, DC: American Educational Research Association and Mahwah, NJ: Erlbaum. Ciarocco, N. J., Stohmetz, D. B., & Lewandowski, G. W., Jr. (2010). Exemplar studies for teaching research methodology. Office of Teaching Resources in Psychology, Society for the Teaching of Psychology. Retrieved from http://www.teachpsych.com/otrp/resources/ciarocco10.pdf Cohen, R. J., & Swerdlik, M. E. (2005). Psychological testing and assessment: An introduction to tests and measurement (6th ed.). Boston, MA: McGraw-Hill. Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229–238. Colosi, L. (1997). Reliability and validity: What’s the difference? Retrieved from http://www.socialresearchmethods.net/tutorial/Colosi/lcolosi2.htm Combs, P. (2000). Major in success: Make college easier, fire up your dreams, and get a very cool job. Converse, J. M. & Presser, S. (1986). Survey questions: Handcrafting the standardized ques- tionnaire. In M.S. Lewis-Beck (Series Ed.), Quantitative applications in the soc ial sciences (Vol. 63). Newbury Park, CA: Sage Publications. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally. Cooper, H. M., & Lindsay, J. J. (1998). Research synthesis and meta-analysis. In L. Bick- man & D. J. Rog (Eds.), Handbook of applied social research methods (pp. 315–337). Thousand Oaks, CA: Sage Publications. Courtney, K. L., & Craven, C. K. (2005). Factors to weigh when considering ele ctronic data collection. Canadian Journal of Nursing Research, 37, 150–159. lan66845_10_ref_p271-294.indd 275 4/20/12 2:51 PM REFERENCES Cox, B. D., Cullen, K. L., Buskist, W., & Benassi, V. A. (2010). Helping undergraduates make the transition to graduate school. In D. S. Dunn, B. C. Beins, M. A. McCar- thy, & G. W. Hill, IV (Eds.), Best practices for teaching beginnings and endings in the psychology major (pp. 319–329). New York, NY: Oxford. Crawford, H. J., & Christensen, L. B. (1995). Surveys. Developing research skills: A labora- tory manual (3rd ed., pp. 55–64). Boston, MA: Allyn & Bacon. Creative Research Systems. (2009). Survey design. Retrieved from http://www.surveysystem.com/sdesign.htm Crites, J. O. (1978). Career Maturity Inventory. Monterey, CA: CTB/McGraw-Hill. Crombag, H. F. M., Wagenaar, W. A., & van Koppen, P. J. (1996). Crashing memories and the problem of ‘source monitoring.’ Applied Cognitive Psychology, 10, 95–104. Crosbie, J. (199). Statistical inference in behavior analysis: Useful friend. The Behavior Analyst, 22, 105–108. Crowne, D. P., & Marlow, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349–354. Dale, A. (2006). Quality issues with survey research. International Journal of Social Research Methodology, 9, 143–158. doi:10.1080/13645570600595330 Daley, E. M., McDermott, R. J., McCormack Brown, K. R., & Kittleson, M. J. (2003). Con- ducting web-based survey research: A lesson in Internet designs. American Journal of Health Behavior, 27, 116–124. Dattilo, J., & Nelson, G. D. (1986). Single-subject evaluation in heal th education. Health Education Quarterly, 13, 249–259. de Leeuw, E. D., & Hox, J. J. (2008). Self-administered questionnaires: Mail surveys and other applications. In E. D. Leeuw, J. J. Hox, & D. A. Dillman (Eds.), International handbook of survey methodology (pp. 239–263). New York, NY: Taylor & Francis/ Erlbaum. DiLalla, D. L., & Dollinger, S. J. (2006). Cleaning up data and running preliminary analy – ses. In F. T. L. Leong & J. T. Austin (Eds.), The psychology research guidebook: A guide for graduate students and research assistants (2nd ed) (pp. 241–253). Thousand Oaks, CA: Sage. Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode sur- veys: The tailored design method (3rd ed.). Hoboken, NJ: Wiley. Dunn, D. (2011). A short guide to writing about psychology (3rd ed.). Boston, MA: Longman/Pearson. lan66845_10_ref_p271-294.indd 276 4/20/12 2:51 PM REFERENCES Dunn, D. S., & Smith, R. A. (2008). Writing as critical thinking. In D. S. Dunn, J. S. Halonen, & R. A. Smith (Eds.), Teaching critical thinking in psychology: A handbook of best practices (pp. 163–173). Malden, MA: Blackwell. Duriau, V. J., Reger, R. K., & Pfarrer, M. D. (2007). A content analysis of the content analy – sis literature in organizational studies. Organizational Research Methods, 10, 5–34. doi:10.1177/1094428106289252 Dutka, S., & Frankel, L.R. (1993). Measurement errors in organizational surveys. Ameri- can Behavioral Scientist, 36, 472–484. Ebbinghaus, H. (1885). Über das Gedächtnis: Untersuchungen zur experimentellen Psychologi e [Translated title: Concerning memory: A contribution to experimental psychol – ogy]. Leipzig, Germany: Duneker & Humblot. Edwards, J. E., & Thomas, M. D. (1993). The organizational survey process: General steps and practical considerations. American Behavioral Scientist, 36, 419–442. Epstein, E. E., McCrady, B. S., Morgan, T. J., Cook, S. M., Kugler, G., & Ziedonis, D. (2007). The successive cohort design: A model for developing new behavioral therapies for drug use disorders, and the application to behavioral couple treat – ment. Addictive Disorders & Their Treatment, 6, 1–19. eSurveyPro. (2009). Questionnaire development. Retrieved from http://www.esurveypro.com/article-questionnaire-development.aspx Fancher, L. J. (2004). The limitations of science. Retrieved from http://www.cod.edu/people/faculty/fancher/Limits.htm Fernald, D. (1984). The Hans legacy: A story of science. Hillsdale, NJ: Erlbaum. Festinger, D. S., & DeMatteo, D. (2008). Subject selection. In A. M. Nezu & C. M. Nezu (Eds.), Evidence-based outcome research: A practical guide to conducting randomized controlled trials for psychosocial interventions (pp. 135–153). New York, NY: Oxford University Press. Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row, Peterson. Festinger, L., & Carlsmith, J. M. (1959). Cognitive consequences of forced compliance. Journal of Abnormal and Social Psychology, 58, 203-210. Fife-Schaw, C. (2000). Surveys and sampling issues. In G. M. Breakwell, S. Hammond, & C. Fife-Schaw (Eds.), Research methods in psychology (2nd ed.) (pp. 88–104). Thousand Oaks, CA: Sage Publications. Fink, A. (1995). How to analyze survey data. Thousand Oaks, CA: Sage Publications. Fink, A., & Kosecoff, J. (1985). How to conduct surveys: A step-by-step guide. Newbury Park, CA: Sage Publications. lan66845_10_ref_p271-294.indd 277 4/20/12 2:51 PM REFERENCES Finley, C. L., & Cowley, B. J. (2005). The effects of a consistent sleep schedule on time taken to achieve sleep. Clinical Case Studies, 4, 304–311. doi:10.1177/1534651003259743 Folkman, S. (2000). Privacy and confidentiality. In B. D. Sales & S. Folkman (Eds.), Ethics in research with human participants (pp. 49–57). Washington, DC: American Psy- chological Association. Forer, B. R. (1949). The fallacy of personal validation: A classroom demonstration of gull – ibility. Journal of Abnormal and Social Psychology, 44, 118–123. Foster, L. H., Watson, T. S., Meeks, C., & Young, J. S. (2002). Single-subject research design for school counselors: Becoming an applied researcher. Professional School Counseling, 6, 146–154. Fowler, F. J., Jr. (1988). Survey research methods (Rev. ed.). In L. Bickman & D.J. Rog (Series Eds.), Applied social research methods series (Vol. 1). Newbury Park, CA: Sage Publications. Freeman, K. A. (2003). Single subject designs. In J. C. Thomas & M. Hersen (Eds.) , Understanding research in clinical and counseling psychology (pp. 181–208). Mahwah, NJ: Erlbaum. Freeman, K. A., & Mash, E. J. (2008). Single-case research designs. In M. Hersen & A. M. Gross (Eds.), Handbook of clinical psychology (Vol. 2): Children and adolescents (pp. 322–350). Hoboken, NJ: Wiley. Frey, J. H., & Oishi, S. M. (1995). How to conduct interviews by telephone and in person. Thousand Oaks, CA: Sage Publications. Futrell, D. (1994, April). Ten reasons why surveys fail. Quality Progress, 65–69. Garson, G. D. (2008). Case studies. Retrieved from http://faculty.chass.ncsu.edu/garson/PA765/cases.htm Garson, G. D. (2008). Time series analysis. Retrieved from http://faculty.chass.ncsu.edu/garson/PA765/time.htm Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8. Glass, G. V., & Smith, M. L. (1979). Meta-analysis of research on class size and achieve – ment. Educational Evaluation and Policy Analysis, 1, 2–16. Gliner, J. A., Morgan, G. A., & Harmon, R. J. (2000). Single-subject designs. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 1327–1329. Gliner, J. A., Morgan, G. A., & Harmon, R. J. (2001). Measurement reliability. Journal of the American Academy of Child Adolescent Psychiatry, 40, 486–488. lan66845_10_ref_p271-294.indd 278 4/20/12 2:51 PM REFERENCES Gliner, J. A., Morgan, G. A., & Harmon, R. J. (2002). Single-factor repeated-measures designs: Analysis and interpretation. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 1014–1016. Goldie, J., Schwartz, L., McConnachie, A., & Morrison, J. (2001). Impact of a new course on students’ potential behavior on encountering ethical dilemmas. Medical Educa- tion, 35, 295–302. Goodlet, C. (2001). Within-subjects designs (repeated measures). Retrieved from http://www.psynt.iupui.edu/Users/cgoodlet/B311/withinsubs.htm Gottschalk, K., & Hjortshoj, K. (2004). The elements of teaching writing: A resource for instructors in all disciplines. Boston: Bedford/St. Martin’s. Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planning miss- ing data designs in psychological research. Psychological Methods, 11, 323–343. doi:10.1037/1082-989X.11.4.323 Gravetter, F. J., & Wallnau, L. B. (2005). Essentials of statistics for the behavioral sciences (5th ed.). Belmont, CA: Thomson/Wadsworth. Green, C. D. (1992). Of immortal mythological beasts: Operationism in ps ychology. Theory & Psychology, 2, 291-320. Greene, M. M. (2003). Program evaluation. In J. C. Thomas & M. Hersen (Eds.), Under- standing research in clinical and counseling psychology (pp. 209–242). Mahwah, NJ: Erlbaum. Gribbons, B., & Herman, J. (1997). True and quasi-experimental designs. Practical Assess- ment, Research & Evaluation, 5. Retrieved from http://pareonline.net/getvn.asp?v=5&n=14 Gueorguieva, R., & Krystal, J. H. (2004). Move over ANOVA: Progress in analyzing repeated-measures data and its reflection in papers published in the Archives of General Psychiatry. Archives of General Psychiatry, 61, 310–317. Hadzi-Pavlovic, D. (1986). Some statistics for analysing change in psy chiatric research. Australian and New Zealand Journal of Psychiatry, 20, 438–451. Haines, C., & Landrum, R. E. (2008). Faculty perceptions of undergraduate students’ par – ticipation in regional psychology conferences. Psychological Reports, 103, 426–430. Hall, J. D., Jones, C. H., & Claxton, A. F. (2008). Evaluation of the Stop & Think social skills program with kindergarten students. Journal of Applied School Psychology, 24, 265–283. doi:10.1080/15377900802093280 Hall, R. (1998). Within-subjects designs. Retrieved from ht t p://we b. m s t .e du/~p s y wo rld/w it h i n _ s u b j e c t s . ht m lan66845_10_ref_p271-294.indd 279 4/20/12 2:51 PM REFERENCES Hanson, S. L., Guenther, R., Kerkhoff, T., & Liss, M. (2000). Ethics: Historical founda – tions, basic principles, and contemporary issues. In R. G. Frank & T. R. Elliott (Eds.), Handbook of rehabilitation psychology (pp. 629–643). Washington, DC: Ameri- can Psychological Association. Harmon, R. J., & Morgan, G. A. (1999). Research problems and variables. Journal of the American Academy of Child Adolescent Psychiatry, 38, 784–785. Harris, B. (1979). Whatever happened to Little Albert? American Psychologist, 34, 151–160. Harris, R. A. (2005). Using sources effectively: Strengthening your writing and avoiding plagiarism (2nd ed.). Glendale, CA: Pyrczak. Hart, P., Eaton, L., Buckner, M., Morrow, B. N., Barrett, D. T., Fraser, D. D., Hooks, D., & Sharrer, R. L. (2008). Effectiveness of a computer-based educational program on nurses’ knowledge, attitude, and skill level related to evidence-based practice. Worldviews on Evidence-Based Nursing, 5, 75–84. Hedges, L. V. (1990). The future of meta-analysis. In K. W. Wachter & M. L. Straf (Eds.), Directions for future methodology (pp. 11–26). New York, NY: Russell Sage Foundation. Hellier, E. J. (1998). Within-subject designs. In J. Nunn (Ed.), Laboratory psychology: A beginner’s guide (pp. 39–58). Hove, England: Psychology Press. Henerson, M. E., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to measure attitudes. Newbury Park, CA: Sage Publications. Henry, G. T. (1990). Practical sampling. In L. Bickman & D.J. Rog (Series Eds.) , Applied social research methods (vol. 21). Newbury Park, CA: Sage Publications. Hettich, P. (2010). Helping undergraduates transition to the workplace. In D. S. Dunn, B. C. Beins, M. A. McCarthy, & G. W. Hill, IV (Eds.), Best practices for teaching begin- nings and endings in the psychology major (pp. 299–318). New York, NY: Oxford. Hettich, P. I., & Helkowski, C. (2005) Connect college to career: A student’s guide to work and life transitions. Belmont, CA: Thomspon Wadsworth. Hetzroni, O. E., & Shrieber, B. (2004). Word processing as an assistive technology tool for enhancing academic outcomes of students with writing disabilities in the general classroom. Journal of Learning Disabilities, 37, 143–154. Hoffer, A. (1999). The public reaction to double-blind controlled clinical trials. Journal of Orthomolecular Medicine, 14, 179–184. Holland, J. L. (1958). A personality inventory employing occupational titles. Journal of Applied Psychology, 42, 336–342. Holland, J. L. (1959). A theory of vocational choice. Journal of Counseling Psychology, 6, 35–45. lan66845_10_ref_p271-294.indd 280 4/20/12 2:51 PM REFERENCES Holland, J. L. (1973). Making vocational choices: A theory of careers. Holland, J. L. (1994). Self-Directed Search® (SDS®) Form R (4th ed.). [Instrument]. Odessa, FL: Psychological Assessment Resources. Horn, W. F., & Heerboth, J. (1982). Single-case experimental designs and program evalu – ation. Evaluation Review, 6, 403–424. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179. Hosie, T. (1994). Program evaluation: A potential area of expertise for counselors. Coun- selor Education & Supervision, 33, 349–355. HR-Survey. (2008). Constructing or writing items for a survey. Retrieved from http://www.hr-survey.com/ItemConstruction.htm Hume, K., & Odom, S. (2007). Effects of an individual work system on the independent functioning of students with autism. Journal of Autism and Developmental Disor- ders, 37, 1166–1180. doi:10.1007/s10803-006-0260-5 Hunter, J. E., Schmidt, F. L., & Hunter, R. (1979). Differential validity of employment tests by race: A comprehensive review and analysis. Psychological Bulletin, 86, 721–735. Instructions in Regard to Preparation of Manuscript. (1929). Psychological Bulletin, 26, 57–63. International Directory of Company Histories. (2001). The Gallup organization (Vol. 37). Chicago, IL: St. James Press. Ivancevich, J. M. (1974). Changes in performance in a management by ob jectives pro – gram. Administrative Science Quarterly, 19, 563–574. Jackson, D. N. (1970). A sequential system for personality scale development. In C. D. Spielberger (Ed.), Current topics in clinical and community psychology (Vol. 2, pp. 61–96). New York, NY: Academic Press. Jackson, J. S., & Antonucci, T. C. (1994). Survey methodology in life-span human devel – opment research. In S. H. Cohen & H. W. Reese (Eds.), Life-span developmental psychology: Methodological contributions (pp. 65–94). Hillsdale, NJ: Erlbaum. Jamison, D. (2008, May 30). George Gallup’s gaffe. Retrieved from http://www.suite101.com/content/george-gallup-gaffe-a55345 Jenson, W. R., Clark, E., Kircher, J. C., & Kristjansson, S. D. (2007). Statistical reform: Evidence-based practice, meta-analyses, and single subject designs. Psychology in the Schools, 44, 483–493. doi:10.1002/pits.20240 lan66845_10_ref_p271-294.indd 281 4/20/12 2:51 PM REFERENCES Jones, E. E. (1976). Forward. In R. A. Wicklund & J. W. Brehm (Eds.), Explorations in cogni- tive dissonance (p. x). Hillsdale, NJ: Erlbaum. Kanter, J. W., Kohlenberg, R. J., & Loftus, E. F. (2002). Demand characteristics, treatment rationales, and cognitive therapy for depression. Prevention & Treatment, 5 (41). Karcen, A. C., & Wallace, I. J. (Eds.). (2008). Applying to graduate school in psychology: Advice from successful students and prominent psychologists. Washington, DC: American Psychological Association. Kazdin, A. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63, 146–159. doi:10.1037/0003-066X.63.3.146 Keith-Spiegel, P., & Wiederman, M. W. (2000). The complete guide to graduate school admission: Psychology, counseling, and related professions (2nd ed.). Mahwah, NJ: Erlbaum. Keller, J., & Bless, H. (2005). When negative expectancies turn into negativ e performance: The role of ease of retrieval. Journal of Experimental Social Psychology, 41, 535–541. Kempf, A. M., & Remington, P. L. (2007). New challenges for telephone survey research in the twenty-first century. Annual Review of Public Health, 28 , 113–126. doi:10.1146/ annurev.publhealth.28.021406.144059 Keren, G. (1993). Between- or within-subjects design: A methodological dilemma. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 257–272). Hillsdale, NJ: Erlbaum. Kidd, R. F. (1977). Manipulation checks: Some further considerations. Representative Research in Social Psychology, 8, 94–97. Klier, C. M., Geller, P. A., & Neugebauer, R. (2000). Minor depressive disorder in the con – text of miscarriage. Journal of Affective Disorders, 59, 13–21. Koch, S. (1992). Psychology’s Bridgman vs. Bridgman’s Bridgman. Theory & Psychology, 2, 261–290. Korn, J.H. (1988). Students’ roles, responsibilities, and rights as research participants. Teaching of Psychology, 15, 74-78. Kosslyn, S. M. (1994). Elements of graph design. New York, NY: W. H. Freeman. Kuther, T. L., & Morgan, R. D. (2007). Careers in psychology: Opportunities in a changing world (2nd ed). Belmont, CA: Thomson Higher Education. Labovitz, S. (1967). Some observations on measurement and statistics. Social Forces, 46, 151–160. lan66845_10_ref_p271-294.indd 282 4/20/12 2:51 PM REFERENCES Landrum, R. E. (2003). Graduate admission in psychology: Transcripts and the effect of withdrawals. Teaching of Psychology, 30, 323–325. Landrum, R. E. (2008). Undergraduate writing in psychology: Learning to tell the scientific story. Washington, DC: American Psychological Association. Landrum, R. E. (2009). Finding jobs with a psychology bachelor’s degree. Washington, DC: American Psychological Association. Landrum, R. E., & Clark, J. (2005). Graduate admissions criteria in psychol ogy: An update. Psychological Reports, 97, 481–484. Landrum, R. E., & Clark, J. (2006). Student perceptions of textbook outlines. College Stu- dent Journal, 40, 646–650. Landrum, R. E., & Davis, S. F. (2010). The psychology major: Career options and strategies for success (4th ed.). Upper Saddle River, NJ: Pearson Prentice-Hall. Landrum, R. E., & Harrold, R. (2003). What employers want from psychology graduates. Teaching of Psychology, 30, 131–133. Landrum, R. E., & Nelsen, L. R. (2002). The undergraduate research assistantship: An analysis of the benefits. Teaching of Psychology, 29, 15–19. Landrum, R. E., Meliska, C. J., & Loke, W. H. (1988). Effects of caffeine and task experi- ence on task performance. Psychologia, 31, 91–97. Lawler, E. E., III, & Hackman, J. R. (1969). Impact of employee participatio n in the devel – opment of pay incentive plans: A field experiment. Journal of Applied Psychology, 53, 467–471. Leahey, E., Entwisle, B., & Einaudi, P. (2003). Diversity in everyday research prac – tice: The case of data editing. Sociological Methods & Research, 3, 63–89. doi:10.1177/0049124103253461 Lederman, L. C. (1992). Debriefing: Toward a system assessment of theory and practice. Simulation & Gaming, 23, 145–160. Lesik, S. A. (2006). Applying the regression-discontinuity design to infer causality with non-random assignment. Review of Higher Education, 30, 1–19. Levine, H. G. (1985). Principle of data storage and retrieval for use in qualitative evalua – tions. Educational Evaluation and Policy Analysis, 7, 169–186. Levine, M. (1974). Scientific method and the adversary model: Some p reliminary thoughts. American Psychologist, 29, 661–677. lan66845_10_ref_p271-294.indd 283 4/20/12 2:51 PM REFERENCES Light, J. (2010, October 11). Psych majors aren’t happy with options. Wall Street Journal. Retrieved from http://online.wsj.com/article/SB10001424052748704011904575538561813341020.html Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22, 5–55. Litwin, M.S. (1995). How to measure survey reliability and validity. Thousand Oaks, CA: Sage Publications. Lodzinski, A., Motomura, M. S., & Schneider, F. W. (2005). Intervention and evaluation. In F. W. Schneider, J. A. Gruman, & L. M. Coutts (Eds.), Applied social psychology: Understanding and addressing social and practical problems (pp. 55–73). Thousand Oaks, CA: Sage Publications. Loftus, E. F. (1975). Leading questions and the eyewitness report. Cognitive Psychology, 7, 560–572. Loftus, E. F., & Hoffman, H. G. (1989). Misinformation and memory: The creation of new memories. Journal of Experimental Psychology: General, 118, 100–104. Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An exam – ple of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13, 585–589. Lohr, S. L. (2008). Coverage and sampling. In E. D. Leeuw, J. J. Hox, & D. A. Dillman (Eds.), International handbook of survey methodology (pp. 97–112). New York, NY: Taylor & Francis/Erlbaum. MacKenzie, I. S. (2008). Research note: Within-subjects vs. between-subjects designs: Which to use? Retrieved from http://www.yorku.ca/mack/RN-Counterbalancing.html Mallon, S. D., Kingsley, D., Affleck, G., & Tennen, H. (1998). Methodological trends in Journal of Personality: 1970–1995. Journal of Personality, 66, 671–685. Mark, M. M., Reichardt, C. S., & Sanna, L. J. (2000). Time-series design and analysis. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 353–389). San Diego, CA: Academic Press. Marlow, D., & Crowne, D. P. (1961). Social desirability and response to perceived situ – ational demands. Journal of Consulting Psychology, 25, 109–115. Martin, G. L., Thompson, K., & Regehr, K. (2004). Studies using single-subject designs in sport psychology: 30 years of research. Behavior Analyst, 27, 263–280. Martz, E., Bodner, T., & Livneh, H. (2009). Coping as a moderator of disability and psy- chosocial adaptation among Vietnam theater veterans. Journal of Clinical Psychol- ogy, 65, 94–112. doi:10.1002/jclp.20541 lan66845_10_ref_p271-294.indd 284 4/20/12 2:51 PM REFERENCES Masters, J. L. (2005). A Midwestern response to the events of September 11, 2001. Journal of Loss and Trauma, 10, 369–381. doi:10.1080/15335020590956800 Maxwell, S. E., & Delaney, H. D. (2004). Designing experiments and analyzing data: A model comparison perspective (2nd ed.). Mahwah, NJ: Erlbaum. McCartney, K., & Weiss, H. B. (2007). Data for a democracy: The evolving role of evaluation in policy and program development. In J. L. Aber, S. J. Bishop-Josef, S. M. Jones, K. T. McLearn, & D. A. Phillips (Eds.), Child development and social policy: Knowledge for action (pp. 59–76). Washington, DC: American Psychological Association. McDevitt, P. K., & Small, M. H. (2002). Proprietary market research: Are online panels appropriate? Marketing Intelligence & Planning, 20, 285–296. doi:10.1108/02634500210441530 McGreevy, J. (2008). Common mistakes in writing survey items. Retrieved from http://www.gpaea.k12.ia.us/media/2828/common%20mistakes%20writing%20 survey%20items.pdf McReynolds, L. V., & Thompson, C. K. (1986). Flexibility of the single-subject experim en – tal designs. Part I: Review of the basics of single-subject designs. Journal of Speech and Hearing Disorders, 51, 194–203. Meltzoff, J. (1998). Critical thinking about research: Psychology and related fields. Washington, DC: American Psychological Association. Meltzoff, J. (1998). Research designs and threats to internal validity. In Critical thinking about research: Psychology and related fields (pp. 83–100). Washington, DC: Ameri- can Psychological Association. Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychol- ogy, 67, 371–378. Milgram, S. (1965). Some conditions of obedience and disobedience to a uthority. Human Relations, 18, 57–76. Moeller, F. G., Barratt, E. S., Dougherty, D. M., Schmitz, J. M., & Swann, A. C. (2001). Psy- chiatric aspects of impulsivity. American Journal of Psychiatry, 158, 1783–1793. Morgan, B. L., & Korschgen, A. J. (2009). Majoring in psych? Career options for psychology undergraduates (4th ed.). Needham Heights, MA: Allyn & Bacon. Morgan, D. L. (2009). Using single-case design and personalized behavior change projects to teach research methods. Teaching of Psychology, 36, 267–269. doi:10.1080/00986280903175715 Morgan, G. A., Gliner, J. A., & Harmon, R. J. (2000). Quasi-experimental designs. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 794–796. lan66845_10_ref_p271-294.indd 285 4/20/12 2:51 PM REFERENCES Morgan, G. A., Gliner, J. A., & Harmon, R. J. (2001). Measurement validity. Journal of the American Academy of Child Adolescent Psychiatry, 40, 729–731. Morgan, G. A., Gliner, J. A., & Harmon, R. J. (2002). Selection of inferential statistics: An overview. Journal of the American Academy of Child Adolescent Psychiatry, 41, 478–481. Morris, S. (1985). The genes from the genies: The nature of theory in science. Unpublished manuscript. Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and appli- cations. Thousand Oaks, CA: Sage Publications. Newell, P. (2005). The Galilean library. Retrieved from ht t p://w w w.galilea n-librar y.org/ma nuscript.php?post id=43833 Newman, F. L., & Wong, S. E. (2004). Progress and outcomes assessment in individual patient data: Selecting single-subject design and statistical procedures. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes assessment: Volume 1: General considerations (pp. 273–289). Mahwah, NJ: Erlbaum. Newsom, J. T. (2006). Distinguishing between random and fixed: Variables, effects, and coeffi- cients. Retrieved from http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0C CkQFjAA&url=htt p%3A%2F%2Fwww.upa.pdx.edu%2FIOA%2Fnewsom%2Fmlrcl ass%2Fho_randfixd.doc&ei=FEaPT7PQK4WMigKWodHwAg&usg=AFQjCNFRzl OsjEuf NnvLAB8-RJWt42786A Newsom, J. T. (n.d.). Factorial ANOVA. Retrieved from ht t p://w w w.upa.pdx.edu/IOA/pa551/lect u r12.ht m Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135, 151–165. Nicol, A. A. M., & Pexman, P. M. (2010). Displaying your findings: A practical guide for creat- ing figures, posters, and presentations (6th ed.). Washington, DC: American Psycho- logical Association. Nicolle, P. S., & Lou, Y. (2008). Technology adoption into teaching and learning by mainstream university faculty: A mixed methodology study revealing the “how, when why, and why not.” Journal of Educational Computing Research, 39, 235–265. doi:10.2190/EC.39.3.c Niland, N. (2007). Thanks for the memories! Retrieved from http://www.nniland.com/AP%20Psych%20Documents/Summer%20Reading%20 Article%20-%20Loftus.pdf Norcross, J. C., Kohout, J. L., & Wicherski, M. (2006). Graduate admissions in psychol – ogy: I. The application process. Eye on Psi Chi, 10(2), 28-29, 42–43. lan66845_10_ref_p271-294.indd 286 4/20/12 2:51 PM REFERENCES Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). E-research: Ethics, security, design, and control in psychological research on the Internet. Journal of Social Issues, 58, 161–176. Nugent, W. R. (2009). Meta-analysis as a research synthesis methodology: Cause for con – cern. Journal of Social Service Research, 35 , 181–192. doi:10.1080/01488370802678967 Olive, M. L., & Smith, B. W. (2005). Effect size calculations and single subject designs. Educational Psychology, 25, 313–324. doi:10.1080/0144341042000301238 Ottenbacher, K. J. (1990). Visual inspection of single-subject data: An empirical analysis. Mental Retardation, 28, 283–290. Packer, D. J. (2008). Identifying systematic disobedience in Milgram’s ob edience experi – ments: A meta-analytic review. Perspectives on Psychological Science, 3, 301–304. Page-Bucci, H. (2003). The value of Likert scales in measuring attitudes of online learners. Retrieved from http://www.hkadesigns.co.uk/websites/msc/reme/likert.htm Pates, J., Maynard, I., & Westbury, T. (2001). An investigation into the effects of hyp- nosis on basketball performance. Journal of Applied Sport Psychology, 13, 84–102. doi:10.1080/10413200109339005 Paul, D. B., & Blumenthal, A. L. (1989). On the trail of Little Albert. The Psychological Record, 39, 547–553. Pearson Education, Inc. (2007). Minnesota Multiphasic Personality Inventory-2. Retrieved from http://www.pearsonassessments.com/tests/mmpi_2.htm Pelham, W. E., Jr., Wheeler, T., & Chronis, A. (1998). Empirically supported psychosocial treatments for Attention Deficit Hyperactivity Disorder. Journal of Child Clinical Psychology, 27, 190–205. Plonsky, M. (2006). Psychology with style: A hypertext writing guide. Retrieved from the Uni- versity of Wisconsin-Stevens Point from ht t p://w w w.uwsp.edu/psyc h/apa4b.ht m Plucker, J. (2007). Human intelligence: Hermann Ebbinghaus. Retrieved from Indiana Uni- versity, http://www.indiana.edu/~intell/ebbinghaus.shtml Ponterotto, J. G., & Ruckdeschel, D. E. (2007). An overview of coefficient alpha and a reliability matrix for estimating the adequacy of internal consistency co efficients with psychological research measures. Perceptual and Motor Skills, 105, 997–1014. doi:10.2466/PMS.105.3.997-1014 Pope, C., & Mays, N. (2006). Observational methods. In C. Pope & N. Ma ys (Eds.), Qualita- tive methods in health research (pp. 1–11). Williston, VT: Malden Blackwell Publishing. Price, I. (2000). Between groups vs. repeated measures. Retrieved from http://www.une.edu.au/WebStat/unit_materials/c2_research_design/between_ repeated_measure.htm lan66845_10_ref_p271-294.indd 287 4/20/12 2:51 PM REFERENCES Price, M. (2002). Beyond “Gotcha!”: Situating plagiarism in poli cy and pedagogy. College Composition and Communication, 54, 88–115. Pryjmachuk, S., & Richards, D. A. (2007). Look before you leap and don’t put all your eggs in one basket: The need for caution and prudence in quantitative data analy – sis. Journal of Research in Nursing, 12, 43–54. doi:10.1177/1744987106070260 Psi Chi. (2009). Benefits & advantages of membership. Retrieved from http://www.psichi.org/about/benefits.aspx Psychological Assessment Resources. (2001). Welcome to the self-directed search. Retrieved from http://www.self-directed-search.com/ Rajecki, D. W., & Borden, V. M. H. (2009). First-year employment outcomes of US psy- chology graduates revisited: Need for a degree, salary, and relatedness to the major. Psychology Learning and Teaching, 8, 23–29. Rapoff, M., & Stark, L. (2008). Editorial: Journal of Pediatric Psychology statement of pur – pose: Section on single-subject studies. Journal of Pediatric Psychology, 33, 16–21. doi:10.1093/jpepsy/jsm101 Rattray, J., & Jones, M. C. (2007). Essential elements of questionnaire design and develop – ment. Journal of Clinical Nursing, 16, 234–243. doi:10.1111/j.1365-2702.2006.01573.x Reeves, B., & Geiger, S. (1994). Designing experiments that assess psychological responses to media messages. In A. Lang (Ed.), Measuring psychological responses to media messages (pp. 165–180). Hillsdale, NJ: Erlbaum. Rind, B., & Strohmetz, D. (1999). Effect on restaurant tipping of a helpful message writ – ten on the back of customers’ checks. Journal of Applied Social Psychology, 29, 139–144. doi:10.1111/j.1559-1816.1999.tb01378.x Riniolo, T. C., Koledin, M., Drakulic, G. M., & Payne, R. A. (2003). An archival study of eyewitness memory of the Titanic’s final plunge. Journal of General Psychology, 130, 89–95. Roberts, J. S., Laughlin, J. E., & Wedell, D. H. (1999). Validity issues in the Likert and Thurstone approaches to attitude measurement. Educational and Psychological Measurement, 59, 211–233. Robison, F. F., Morran, D. K., & Hulse-Killacky, D. (1989). Single-subject research designs for group counselors studying their own groups. Journal for Specialists in Group Work, 14, 93–97. Roelen, C. A. M., Koopmans, P. C., & Groothoff, J. W. (2008). Which work factors deter – mine job satisfaction? Work, 30, 433–439. Roethlisberger, F. J., & Dickson, W. J. (1939). Management and the worker. Cambridge, MA: Harvard University Press. lan66845_10_ref_p271-294.indd 288 4/20/12 2:51 PM REFERENCES Roig, M. (2008, Winter). Avoiding those little inadvertent lies when writing papers. Eye on Psi Chi, 12(2), 31–33. Rose, D. S., & Fiore, K. E. (1999). Practical considerations and alternative research meth – ods for evaluating HR programs. Journal of Business and Psychology, 14, 235–251. Rosenthal, R. (1966). Experimenter effects in behavioral research. East Norwalk, CT: Apple- ton Century Crofts. Rosenthal, R. (1967). Covert communication in the psychological experi ment. Psychologi- cal Bulletin, 76, 356–367. Rosenthal, R., & Rosnow, R. L. (1975). The volunteer subject. New York, NY: Wiley. Rudas, T. (2005). Mixture models of missing data. Quality & Quantity, 39, 19–36. Rushton, J.P. (1988). Race differences in behavior: A review of evolutionary analysis. Journal of Personality and Individual Differences, 9, 1035–1040. Rutter, M. (2007). Proceeding from observed correlation to causal inference: The use of natural experiments. Perspectives on Psychological Science, 2, 377–395. Salovey, P. (2000). Results that get results: Telling a good story. In R. J. Sternberg (Ed.), Guide to publishing in psychology journals (pp. 121–132). Cambridge, England: Cambridge University Press. Sanford, E. C. (1914). Psychic research in the animal field: Der Kluge Hans and the Elber – feld horses. American Journal of Psychology, 25, 1–31. Sayette, M. A., Mayne, T. J., & Norcross, J. C. (2010). Insider’s guide to graduate programs in clinical and counseling psychology (2010/2011 edition). New York, NY: Guilford Press. Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378–397. Schultheiss, D. E. P. (2008). Psychology as a major: Is it right for me and what can I do with my degree? Washington, DC: American Psychological Association. Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psy- chologist, 54, 93–105. Scott, J. M., Koch, R. E., Scott, G. M., & Garrison, S. M. (1999). Psychology student writer’s manual. Upper Saddle River, NJ: Prentice Hall. Seashore, S. E. (1987). Surveys in organizations. In G. Salvendy (Ed.), Handbook of human factors (pp. 313–328). New York, NY: John Wiley & Sons. Seligman, M. E. P. (1980). Harris on selective misrepresentation: A selective misrepresen – tation of Seligman. American Psychologist, 35, 214–215. lan66845_10_ref_p271-294.indd 289 4/20/12 2:51 PM REFERENCES Seymour, E., Hunter, A.-B., Laursen, S. L., & Deantoni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. Science Education, 88, 493–534. doi:10.1001/sce.10131 Shadish, W. R., & Cook, T. D. (2009). The renaissance of field experimentation in evaluat – ing interventions. Annual Review of Psychology, 60, 607–629. Shadle, S. (2006, December). Writing and citing workshop. Presented by the Center for Teaching and Learning, Boise State University, Boise, ID. Shakow, D. (1930). Hermann Ebbinghaus. American Journal of Psychology, 42, 505–518. Sharpley, C. F. (2003). N = 1 research methodology in counselling: Focusing upon the individual client. Australian Journal of Guidance & Counselling, 13, 133–144. Silvia, P. J., Delaney, P. F., & Marcovitch, S. (2009). What psychology majors could (and should) be doing. Washington, DC: American Psychological Association. Singer, E., & Levine, F. J. (2003). Protection of human subjects of research: Recent devel – opments and future prospects for the social sciences. Public Opinion Quarterly, 67, 148–164. Sleigh, M. J., & Ritzer, D. R. (2007). Undergraduate research experience: Preparation for the job market. Eye on Psi Chi, 11 (3), 27–30. Smith, L. D., Best, L. A., Stubbs, A., Archibald, A. B., & Roberson-Nay, R. (2002). Con- structing knowledge: The role of graphs and tables in hard and soft psychology. American Psychologist, 57, 749–761. doi:10.1037//0003—66X.57.10.749 Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760. Smith, M. U., & Scharmann, L. C. (1999). Defining versus describing the nature of sci – ence: A pragmatic analysis for classroom teachers and science educators. Science Education, 83, 493–509. Smith, R. A. (2000). Documenting your scholarship: Citations and references. In R. J. Sternberg (Ed.), Guide to publishing in psychology journals (pp. 146–157). Cam- bridge, England: Cambridge University Press. Sonpar, K., & Golden-Biddle, K. (2008). Using content analysis to elaborate adoles – cent theories of organization. Organizational Research Methods, 11, 795–814. doi:10.1177/1094428106297804 Spector, P. E. (1992). Summated rating scale construction: An introduction. In M.S. Lewis- Beck (Series Ed.), Quantitative applications in the social sciences (Vol. 82). Newbury Park, CA: Sage Publications. Stagner, R. (1982). The importance of historical context. American Psychologist, 37, 856. lan66845_10_ref_p271-294.indd 290 4/20/12 2:51 PM REFERENCES Stake, R. E. (1994). Case studies. In N. Denzin, & Y. Lincoln (Eds.), Handbook of qualitative research (pp. 236–247). Thousand Oaks, CA: Sage Publications. Stambor, Z. (2008, July/August). Make the most of your post. Monitor on Psychology, 39(7), 80–81. Stanovich, K. E. (2010). How to think straight about psychology (9th ed.). Boston, MA: Allyn & Bacon/Pearson. StatPac, Inc. (2009). Sampling methods. Retrieved from http://www.statpac.com/surveys/sampling.htm StatTrek, Inc. (2009). AP statistics tutorial: Survey sampling methods. Retrieved from http://stattrek.com/AP-Statistics-2/Survey-Sampling-Methods.aspx Sternberg, R. J. (2000). Titles and abstracts: They only sound unimportant. In R. J. Stern – berg (Ed.), Guide to publishing in psychology journals (pp. 37–40). Cambridge, Eng- land: Cambridge University Press. Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103, 677–680. Suarez-Balcazar, Y., Balcazar, F. E., & Taylor-Ritzler, T. (2009). Using the Internet to con – duct research with culturally diverse populations: Challenges and opportunities. Cultural Diversity and Ethnic Minority Psychology, 15, 96–104. doi:10.1037/a0013179 Sue, V. M., & Ritter, L. A. (2007). Conducting online surveys. Los Angeles, CA: Sage. Sussner, B. D., Smelson, D. A., Rodrigues, S., Kline, A., Losonczy, M., & Ziedonis, D. (2006). The validity and reliability of a brief measure of cocaine craving. Drug and Alcohol Dependence, 83, 233–237. doi:10.1016/j.drugalcdep.2005.11.022 Tankersley, M., McGoey, K. E., Dalton, D., Rumrill, P. D., Jr., & Balan, C. M. (2006). Single subject research methods in rehabilitation. Work, 26, 85–92. Taylor, R. L., & Adams, G. L. (1982). A review of single-subject methodologies in applied settings. Journal of Applied Behavioral Science, 18, 95–103. Thomas, J. C., & Selthon, L. (2003). Planning data collection and perf orming analyses. In J. C. Thomas & M. Hersen (Eds.), Understanding research in clinical and counseling psychology (pp. 319–339). Mahwah, NJ: Erlbaum. Thorpe, J., & Ward, T. (2007, April). How to get the most out of a conference. Observer, 20(4), 33–36. Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 26, 249–269. Townsend, J. C. (1953). Introduction to experimental method for psychology and the social sci- ences. New York, NY: McGraw-Hill. lan66845_10_ref_p271-294.indd 291 4/20/12 2:51 PM REFERENCES Tuckel, P., & O’Neill, H. (2002). The vanishing respondent in telephone surveys. Journal of Advertising Research, 42, 26–48. Tufte, E. R. (1983). The visual display of quantitative information. Cheshire, CA: Graphics Press. Underwood, B. J. (1957). Psychological research. New York, NY: Appleton-Century-Crofts. University of Texas at Austin. (2007). Assess teaching: Writing survey questions. Retrieved from http://www.utexas.edu/academic/ctl/assessment/iar/teaching/plan/method/ survey/writing.php Vagias, W. M. (2006). Likert-type scale response anchors. Retrieved from http://www.hehd.clemson.edu/PRTM/TRMcenter/scale.pdf Visser, P. S., Krosnick, J. A., & Lavrakas, P. J. (2000). Survey research. In H. T. Reid & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 223–252). New York, NY: Cambridge University Press. Vowler, S. L. (2007). Analyzing data: Choosing appropriate statistical data. Hospital Phar- macist, 14, 39–44. Wade, C. (1990, January). Critical thinking about critical thinking. Paper presented at the Twelfth Annual National Institute on the Teaching of Psychology, St. Petersburg Beach, FL. Wahlstrom, C., & Williams, B. K. (2004). College to career: Your road to personal success. Mason, OH: South-Western. Walfish, S., & Hess, A. K. (Eds.). (2001). Succeeding in graduate school: The career guide for psychology students. Mahwah, NJ: Erlbaum. Walshe, C. E., Caress, A. L., Chew-Graham, C., & Todd, C. J. (2004). Case studies: A research strategy appropriate for palliative care? Palliative Medicine, 18, 677–684. Walther, J. (2009, May 31). “Dewey defeats Truman.” The True Facts. Retrieved from http://www.thetruefacts.com/dewey-defeats-truman Ward, R. A., & Grasha, A. F. (1986). Using astrology to teach research methods to intro- ductory students. Teaching of Psychology, 13, 143–145. Watson, J. B., & Rayner, R. (1920). Conditioned emotional reactions. Journal of Experimen- tal Psychology, 3, 1–14. Wayment, H. A., & Dickson, K. L. (2008). Increasing student participation in undergrad – uate research benefits students, faculty, and department. Teaching of Psychology, 35, 194–197. doi:10.1080/00986280802189213 lan66845_10_ref_p271-294.indd 292 4/20/12 2:51 PM REFERENCES Wegenek, A. R., & Buskist, W. (2010). The insider ’s guide to the psychology major: Everything you need to know about the degree and profession. Washington, DC: American Psychological Association. West, M. A., & Anderson, N. R. (1996). Innovation in top management teams. Journal of Applied Psychology, 81, 680–693. Whitbourne, S. K. (2010, August 10). The Barnum effect: One personality prediction fits all. Psychology Today: Fulfillment at any age [Blog]. Retrieved from ht t p://w w w.psycholog ytoday.com/blog/f ulfillment-a ny-age/201008/t he-bar num -effect-one-personality-prediction-fits-all Whiteside, U., Pantelone, D. W., Hunter-Reel, D., Eland, J., Kleiber, B., & Larimer, M. (2007). Initial suggestions for supervising and mentoring undergraduate research assistants at large research universities. International Journal of Teaching and Learn- ing in Higher Education, 19, 325–330. Wilson, H. W., & Widom, C. S. (2008). An examination of risky sexual behavior and HIV in victims of child abuse and neglect: A 30-year follow-up. Health Psychology, 27, 149–158. doi:10.1037/0278-6133.27.2.149 Woodard, C., Groden, J., Goodwin, M., & Bodfish, J. (2007). A placebo double-blind pilot study of dextromethorphan for problematic behaviors in children with autism. Autism, 11 , 29–41. doi:10.1177/1362361307070989 Woodworth, R.S. (1938). Experimental psychology. New York, NY: Holt. Woodworth, R.S., & Schlosberg, H. (1954). Experimental psychology (2nd ed.). New York, NY: Holt. Wuensch, K. (2005). What is a Likert scale? And how to you pronounce ‘Likert?’ Retrieved from http://core.ecu.edu/psyc/wuenschk/StatHelp/Likert.htm Wuensch, K. L. (2003). Quasi-experimental design. Retrieved from http://core.ecu.edu/psyc/wuenschk/SPSS/RegDiscont-SPSS.doc Wuensch, K. L. (2005). An introduction to research design. Retrieved from http://core.ecu.edu/psyc/wuenschk/DesignLessons.htm Wuensch, K. L. (2007). An introduction to within-subjects analysis of variance. Retrieved from http://core.ecu.edu/psyc/wuenschk/MV/RM-ANOVA/ws-anova.doc Xu, Y., Gelfer, J. I., Sileo, N., Filler, J., & Perkins, P. G. (2008). Effects of peer tutoring on young children’s social interactions. Early Child Development and Care, 178, 617– 635. doi:10.1080/03004430600857485 Zagorski, N. (2005). Profile of Elizabeth F. Loftus. Proceedings of the National Academy of Sciences, 102, 13,721–723. doi:10.1073/pnas.0506223102 lan66845_10_ref_p271-294.indd 293 4/20/12 2:51 PM REFERENCES Zimbardo, P. G. (2002). Psychology in the public service. American Psychologist, 57, 431–433. doi:10.1037//0003-066X.57.6-7.431 Zimbardo, P. G. (2004). Does psychology make a significant different in our lives? Ameri- can Psychologist, 59, 339–351. doi:10.1037/0003-066X.59.5.339 lan66845_10_ref_p271-294.indd 294 4/20/12 2:51 PM REFERENCES