Questionnaire DevelopmentTo create psychological measuring tools, test developers initially write as many as twice the number of items that will appear in the final draft of their questionnaires. They

Questionnaire DevelopmentTo create psychological measuring tools, test developers initially write as many as twice the number of items that will appear in the final draft of their questionnaires. They
Scoring PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 2 Scoring Creating the Questionnaire The scoring system of a questionnaire should be thoughtfully considered by a test developer. Each response option should have an assigned value so items can be summed across the instrument. Knowledge -based questionnaires contain correct and incorrect responses. For dichotomous formats, the coding of the responses may be “1” for a correct answer or “0” for the incorrect answer (Rust & Golombok, 2009). In computer scoring, a su bstitute value may be used for incorrect scores so that the value of “0” is excluded. For person -based questionnaires, scores on a continuum may be coded in a forward or reverse direction depending on the allocation of values to response choices. Likewise, scores may fall into categories which can subsequently be interpreted (Rust & Golombok, 2009). Whether using an offline or online survey, the development of a psychometrically sound measuring tool is an important part of the overall testing process. It m ust be appealing and offer psychological benefits for test takers to maintain their interest from start to completion. PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 3 Scoring Creating the Questionnaire References Rust, J., & Golombok, S. (2009). Modern psychometrics: The science of psychological assessment (3rd ed.). New York, NY: Tay lor & Francis. © 201 6 South University
Questionnaire DevelopmentTo create psychological measuring tools, test developers initially write as many as twice the number of items that will appear in the final draft of their questionnaires. They
Piloting the Survey or Questionnaire PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 2 Piloting the Survey or Questionnaire Creating the Questionnaire After constructing the survey instrument, the next step in the survey planning process is a brief exploratory investigation known as a pilot study. As a method used to determine the feasibility of conducting a survey, whether offline or online, the pilot study offers a dress rehearsal of sorts. In the long run, it may save a test developer valuable time as it can effectively identify methods, q uestionnaire items, or other issues, such as with data analysis, that need to be reexamined (Leedy & Ormrod, 2001). An important consideration for the pilot or tryout sample of respondents is to make certain that it is representative of the population for which the measure is intended. Moreover, the size of the sample should be appropriate to ensure that the sample is representative. It’s good to have five to ten respondents for every item of the questionnaire (Cohen & Swerdlik, 2002). If that is not possi ble, Rust and Golombok (2009) suggested that the pilot sample size be approximately equal to the number of test items plus one. The pilot study should be launched under the same conditions as the planned administrations of the test, and all nuances of the testing process should be as similar as possible to those of the planned administrations to guard against the influence of extraneous factors (Cohen & Swerdlik, 2002). The analysis of the data from the pilot study guides the selection of best -fit test ite ms to be included in the next revision of the test. Drafting an item analysis table is a useful strategy for calculating item facility. Item discrimination and identifying the value of distracters in the array of responses are other tasks under the heading of item analysis. The item analysis process allows for the refinement of the rough draft to help boost the psychometric properties of the revised tool (Rust & Golombok, 2009). PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 3 Piloting the Survey or Questionnaire Creating the Questionnaire References Cohen, R., & Swerdlik, M. (2002). Psychological testing and asses sment: An introduction to test and measurement (5th ed.). Boston, MA: McGraw -Hill. Leedy, P., & Ormrod, J. (2001). Practical research planning and design (7th ed.). Upper Saddle River, NJ: Merrill Prentice Hall. Rust, J., & Golombok, S. (2009). Modern psyc hometrics: The science of psychological assessment (3rd ed.). New York, NY: Taylor & Francis. © 201 6 South University
Questionnaire DevelopmentTo create psychological measuring tools, test developers initially write as many as twice the number of items that will appear in the final draft of their questionnaires. They
Writing and Evaluating Test Items PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 2 Writing and Evaluating Test Items Creating the Questionnaire Last week, you were introduced to the topics of reliability and validity and their relationship to test items and testing. In the process of writing test items that are valid, test developers strive to make sure that their items align with the attitude, behavior, ability, or trait under investigation. Valid items measure what they purport to measure and are not disconnected from the objectives of the test (Sue & Ritter, 2013). Once these items have been written, the preliminary draft of the survey or questionnaire is piloted on a representative sample of the population for which the measure is intended. Once the data from the pilot survey is col lected, the overall performance of the test takers on the measure and on each of the items is analyzed. As we have learned, items are evaluated using statistical procedures known as item analysis which may include analysis of item reliability, item validit y, item discrimination, and item difficulty level (Cohen & Swerdlik, 2002). Writing test items can be a taxing process as it requires both creativity and persistence to address each of the criteria necessary for constructing a valid item or question. One of the first concerns of a test creator is the range of difficulty of the items (Gregory, 2013). For tests of ability, achievement, and aptitude, this concern can be addressed by creating items that are differential and progressively become more difficult as the test progresses. Alternatively, mixing less and more difficult items together may be a strategy of choice if one wants to calculate the split -half reliability index score for the test. Another area of concern in the construction of test items is th e homogeneity or heterogeneity of item content (Gregory, 2013). In some cases, items may tap multifaceted constructs and, as such, may have multiple layers. Alternatively, groups of questions may combine to assess multilayered concepts (Sue & Ritter, 2013) . As indicated earlier, it is not uncommon for a test constructor to initially write twice as many questions or items as will eventually be used in the first draft of the questionnaire. Ideally, this pool of questions contains valid items, the content of w hich effectively measures the domain of investigation. If this assumption is violated items may need to be revised or eliminated (Gregory, 2013). Another concern is about the types of responses required by test takers and the meaning that will be attribut ed to test scores (Cohen & Swerdlik, 2002). Items should clearly measure what they purport to measure, be undergirded by theory, and be as specific as possible (Kaplan & Saccuzzo, 2013). Exceptionally long items that are confusing and misleading should be avoided as well as those loaded with jargon (Sue & Ritter, 2013). The art and science of item writing develops over time and with practice. For this reason, test developers should be prepared to commit long hours and a great deal of sweat for the design o f effective test items. PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 3 Writing and Evaluating Test Items Creating the Questionnaire References Cohen, R., & Swerdlik, M. (2002). Psychological testing and assessment: An introduction to test and measurement (5th ed.). Boston, MA: McGraw -Hill. Gregory, R. (2013). Psychological testing: History, principles, and applications (7th ed.). Boston, MA: Pearson. Kaplan, R., & Saccuzzo, D. (2013). Psychological testing: Principles, applications, & issues (8th ed.). Belmont, CA: Wadsworth. Sue, M., & Ritter, L. (2013). Conducting online surveys (2nd ed.). Los Angeles, CA: Sage. © 201 6 South University
Questionnaire DevelopmentTo create psychological measuring tools, test developers initially write as many as twice the number of items that will appear in the final draft of their questionnaires. They
Person -Based Questionnaires PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 2 Person -Based Questionnaires Types of Questionnaires Whereas knowledge -based tests measure ability, achievement, or aptitude, person -based questionnaires measure attitudes, interests, preferences, mood states, and clinical symptoms. Unlike knowledge -based tests, person -based tests are not necessarily hierarc hical and cumulative. In fact, they may be based on the values that people hold regarding attitudes or interests such that a response to any one item does not carry with it a negative or positive valence (Rust & Golombok, 2009). There is no “right” or “wro ng” answer. Test items on person -based questionnaires may be presented in dichotomous or multiple -choice formats. Scores on these measures may fall on points across a continuum running from low to high on a trait or interest. It is also likely that scores may run in the opposite direction such that a low score on a trait or interest may be associated with a high score on the bipolar or opposite trait or interest (Rust & Golombok, 2009). In addition to dimensional approaches to measure personality traits, typological methods offer an alternative style. An example of a typological method for measuring personality traits is the Myers -Briggs Type Indicator (MBTI) which sorts respondents into one of sixteen categories based on the bipolar traits of four dimensi ons of personality (Gregory, 2013). Another method of scaling test items is to use a rating scale in which participants are asked to rate their judgments of themselves or others. Ratings may range from “never justified” to “always justified,” for example, with final test scores summed across all items. Another type of summative rating scale is the Likert scale generally used to measure attitudes. In this format, a respondent may be presented with an array of choices ranging from “agree” to “disagree” or “a pprove” to “disapprove” across a continuum. Responses may be assigned a value such as “1” or “2” and then summed to indicate a person’s overall position on a n interest or trait. PSY3700 Multimedia Assessment and Psychometrics ©20 16 South University 3 Person -Based Questionnaires Types of Questionnaires References Gregory, R. (2013). Psychological testing: History, principles, and applications (7th ed.). Boston, MA: Pearson. Rust, J., & Golombok, S. (2009). Modern psychometrics: The science of psychological assessment (3rd ed.). New York, NY: Taylor & Francis. © 201 6 South University