Criterion–referenced Test Development – Technical and Legal Guidelines for Corporate Training 3e

<p>List of Figures, Tables, and Sidebars xxiii</p>
<p>Introduction: A Little Knowledge Is Dangerous 1</p>
<p>Why Test? 1</p>
<p>Why Read This Book? 2</p>
<p>A Confusing State of Affairs 3</p>
<p>Misleading Familiarity 3</p>
<p>Inaccessible Technology 4</p>
<p>Procedural Confusion 4</p>
<p>Testing and Kirkpatrick s Levels of Evaluation 5</p>
<p>Certification in the Corporate World 7</p>
<p>Corporate Testing Enters the New Millennium 10</p>
<p>What Is to Come. . . 11</p>
<p>Part I: Background: The Fundamentals 13</p>
<p>1 Test Theory 15</p>
<p>What Is Testing? 15</p>
<p>What Does a Test Score Mean? 17</p>
<p>Reliability and Validity: A Primer 18</p>
<p>Reliability 18</p>
<p>Equivalence Reliability 19</p>
<p>Test–Retest Reliability 19</p>
<p>Inter–Rater Reliability 19</p>
<p>Validity 20</p>
<p>Face Validity 23</p>
<p>Context Validity 23</p>
<p>Concurrent Validity 23</p>
<p>Predictive Validity 24</p>
<p>Concluding Comment 24</p>
<p>2 Types of Tests 25</p>
<p>Criterion–Referenced Versus Norm–Referenced Tests 25</p>
<p>Frequency Distributions 25</p>
<p>Criterion–Referenced Test Interpretation 28</p>
<p>Six Purposes for Tests in Training Settings 30</p>
<p>Three Methods of Test Construction (One of Which You Should Never Use) 32</p>
<p>Topic–Based Test Construction 32</p>
<p>Statistically Based Test Construction 33</p>
<p>Objectives–Based Test Construction 34</p>
<p>Part II: Overview: The CRTD Model and Process 37</p>
<p>3 The CRTD Model and Process 39</p>
<p>Relationship to the Instructional Design Process 39</p>
<p>The CRTD Process 43</p>
<p>Plan Documentation 44</p>
<p>Analyze Job Content 44</p>
<p>Establish Content Validity of Objectives 46</p>
<p>Create Items 46</p>
<p>Create Cognitive Items 46</p>
<p>Create Rating Instruments 47</p>
<p>Establish Content Validity of Items and Instruments 47</p>
<p>Conduct Initial Test Pilot 47</p>
<p>Perform Item Analysis 48</p>
<p>Difficulty Index 48</p>
<p>Distractor Pattern 48</p>
<p>Point–Biserial 48</p>
<p>Create Parallel Forms or Item Banks 49</p>
<p>Establish Cut–Off Scores 49</p>
<p>Informed Judgment 50</p>
<p>Angoff 50</p>
<p>Contrasting Groups 50</p>
<p>Determine Reliability 50</p>
<p>Determine Reliability of Cognitive Tests 50</p>
<p>Equivalence Reliability 51</p>
<p>Test–Retest Reliability 51</p>
<p>Determine Reliability of Performance Tests 52</p>
<p>Report Scores 52</p>
<p>Summary 53</p>
<p>Part III: The CRTD Process: Planning and Creating the Test 55</p>
<p>4 Plan Documentation 57</p>
<p>Why Document? 57</p>
<p>What to Document 63</p>
<p>The Documentation 64</p>
<p>5 Analyze Job Content 75</p>
<p>Job Analysis 75</p>
<p>Job Analysis Models 77</p>
<p>Summary of the Job Analysis Process 78</p>
<p>DACUM 79</p>
<p>Hierarchies 87</p>
<p>Hierarchical Analysis of Tasks 87</p>
<p>Matching the Hierarchy to the Type of Test 88</p>
<p>Prerequisite Test 89</p>
<p>Entry Test 89</p>
<p>Diagnostic Test 89</p>
<p>Posttest 89</p>
<p>Equivalency Test 90</p>
<p>Certification Test 90</p>
<p>Using Learning Task Analysis to Validate a Hierarchy 91</p>
<p>Bloom s Original Taxonomy 91</p>
<p>Knowledge Level 92</p>
<p>Comprehension Level 93</p>
<p>Application Level 93</p>
<p>Analysis Level 93</p>
<p>Synthesis Level 93</p>
<p>Evaluation Level 94</p>
<p>Using Bloom s Original Taxonomy to Validate a Hierarchy 94</p>
<p>Bloom s Revised Taxonomy 95</p>
<p>Gagn&eacute; s Learned Capabilities 96</p>
<p>Intellectual Skills 96</p>
<p>Cognitive Strategies 97</p>
<p>Verbal Information 97</p>
<p>Motor Skill 97</p>
<p>Attitudes 97</p>
<p>Using Gagn&eacute; s Intellectual Skills to Validate a Hierarchy 97</p>
<p>Merrill s Component Design Theory 98</p>
<p>The Task Dimension 99</p>
<p>Types of Learning 99</p>
<p>Using Merrill s Component Design Theory to Validate a Hierarchy 99</p>
<p>Data–Based Methods for Hierarchy Validation 100</p>
<p>Who Killed Cock Robin? 102</p>
<p>6 Content Validity of Objectives 105</p>
<p>Overview of the Process 105</p>
<p>The Role of Objectives in Item Writing 106</p>
<p>Characteristics of Good Objectives 107</p>
<p>Behavior Component 107</p>
<p>Conditions Component 108</p>
<p>Standards Component 108</p>
<p>A Word from the Legal Department About Objectives 109</p>
<p>The Certification Suite 109</p>
<p>Certification Levels in the Suite 110</p>
<p>Level A Realworld 110</p>
<p>Level B High–Fidelity Simulation 111</p>
<p>Level C Scenarious 111</p>
<p>Quasi–Certification 112</p>
<p>Level D Memorization 112</p>
<p>Level E Attendance 112</p>
<p>Level F Affiliation 113</p>
<p>How to Use the Certification Suite 113</p>
<p>Finding a Common Understanding 113</p>
<p>Making a Professional Decision 114</p>
<p>The correct level to match the job 114</p>
<p>The operationally correct level 114</p>
<p>The consequences of lower fidelity 115</p>
<p>Converting Job–Task Statements to Objectives 116</p>
<p>In Conclusion 119</p>
<p>7 Create Cognitive Items 121</p>
<p>What Are Cognitive Items? 121</p>
<p>Classification Schemes for Objectives 122</p>
<p>Bloom s Cognitive Classifications 123</p>
<p>Types of Test Items 129</p>
<p>Newer Computer–Based Item Types 129</p>
<p>The Six Most Common Item Types 130</p>
<p>True/False Items 131</p>
<p>Matching Items 132</p>
<p>Multiple–Choice Items 132</p>
<p>Fill–In Items 147</p>
<p>Short Answer Items 147</p>
<p>Essay Items 148</p>
<p>The Key to Writing Items That Match Jobs 149</p>
<p>The Single Most Useful Improvement You Can Make in Test Development 149</p>
<p>Intensional Versus Extensional Items 150</p>
<p>Show Versus Tell 152</p>
<p>The Certification Suite 155</p>
<p>Guidelines for Writing Test Items 158</p>
<p>Guidelines for Writing the Most Common Item Types 159</p>
<p>How Many Items Should Be on a Test? 166</p>
<p>Test Reliability and Test Length 166</p>
<p>Criticality of Decisions and Test Length 167</p>
<p>Resources and Test Length 168</p>
<p>Domain Size of Objectives and Test Length 168</p>
<p>Homogeneity of Objectives and Test Length 169</p>
<p>Research on Test Length 170</p>
<p>Summary of Determinants of Test Length 170</p>
<p>A Cookbook for the SME 172</p>
<p>Deciding Among Scoring Systems 174</p>
<p>Hand Scoring 175</p>
<p>Optical Scanning 175</p>
<p>Computer–Based Testing 176</p>
<p>Computerized Adaptive Testing 180</p>
<p>8 Create Rating Instruments 183</p>
<p>What Are Performance Tests? 183</p>
<p>Product Versus Process in Performance Testing 187</p>
<p>Four Types of Rating Scales for Use in Performance Tests (Two of Which You Should Never Use) 187</p>
<p>Numerical Scales 188</p>
<p>Descriptive Scales 188</p>
<p>Behaviorally Anchored Rating Scales 188</p>
<p>Checklists 190</p>
<p>Open Skill Testing 192</p>
<p>9 Establish Content Validity of Items and Instruments 195</p>
<p>The Process 195</p>
<p>Establishing Content Validity The 196</p>
<p>Single Most Important Step Face Validity 196</p>
<p>Content Validity 197</p>
<p>Two Other Types of Validity 202</p>
<p>Concurrent Validity 202</p>
<p>Predictive Validity 208</p>
<p>Summary Comment About Validity 209</p>
<p>10 Initial Test Pilot 211</p>
<p>Why Pilot a Test? 211</p>
<p>Six Steps in the Pilot Process 212</p>
<p>Determine the Sample 212</p>
<p>Orient the Participants 213</p>
<p>Give the Test 214</p>
<p>Analyze the Test 214</p>
<p>Interview the Test–Takers 215</p>
<p>Synthesize the Results 216</p>
<p>Preparing to Collect Pilot Test Data 217</p>
<p>Before You Administer the Test 217</p>
<p>Sequencing Test Items 217</p>
<p>Test Directions 218</p>
<p>Test Readability Levels 219</p>
<p>Lexile Measure 220</p>
<p>Formatting the Test 220</p>
<p>Setting Time Limits Power, Speed, and Organizational Culture 221</p>
<p>When You Administer the Test 222</p>
<p>Physical Factors 222</p>
<p>Psychological Factors 222</p>
<p>Giving and Monitoring the Test 223</p>
<p>Special Considerations for Performance Tests 225</p>
<p>Honesty and Integrity in Testing 231</p>
<p>Security During the Training–Testing Sequence 234</p>
<p>Organization–Wide Policies Regarding Test Security 236</p>
<p>11 Statistical Pilot 241</p>
<p>Standard Deviation and Test Distributions 241</p>
<p>The Meaning of Standard Deviation 241</p>
<p>The Five Most Common Test Distributions 244</p>
<p>Problems with Standard Deviations and Mastery Distributions 247</p>
<p>Item Statistics and Item Analysis 248</p>
<p>Item Statistics 248</p>
<p>Difficulty Index 248</p>
<p>P–Value 249</p>
<p>Distractor Pattern 249</p>
<p>Point–Biserial Correlation 250</p>
<p>Item Analysis for Criterion–Referenced Tests 251</p>
<p>The Upper–Lower Index 253</p>
<p>Phi 255</p>
<p>Choosing Item Statistics and Item Analysis Techniques 255</p>
<p>Garbage In–Garbage Out 257</p>
<p>12 Parallel Forms 259</p>
<p>Paper–and–Pencil Tests 260</p>
<p>Computerized Item Banks 262</p>
<p>Reusable Learning Objects 264</p>
<p>13 Cut–Off Scores 265</p>
<p>Determining the Standard for Mastery 265</p>
<p>The Outcomes of a Criterion–Referenced Test 266</p>
<p>The Necessity of Human Judgment in Setting a Cut–Off Score 267</p>
<p>Consequences of Misclassification 267</p>
<p>Stakeholders 268</p>
<p>Revisability 268</p>
<p>Performance Data 268</p>
<p>Three Procedures for Setting the Cut–Off Score 269</p>
<p>The Issue of Substitutability 269</p>
<p>Informed Judgment 270</p>
<p>A Conjectural Approach, the Angoff Method 272</p>
<p>Contrasting Groups Method 278</p>
<p>Borderline Decisions 282</p>
<p>The Meaning of Standard Error of Measurement 282</p>
<p>Reducing Misclassification Errors at the Borderline 284</p>
<p>Problems with Correction–for–Guessing 285</p>
<p>The Problem of the Saltatory Cut–Off Score 287</p>
<p>14 Reliability of Cognitive Tests 289</p>
<p>The Concepts of Reliability, Validity, and Correlation 289</p>
<p>Correlation 290</p>
<p>Types of Reliability 293</p>
<p>Single–Test–Administration Reliability Techniques 294</p>
<p>Internal Consistency 294</p>
<p>Squared–Error Loss 296</p>
<p>Threshold–Loss 296</p>
<p>Calculating Reliability for Single–Test Administration Techniques 297</p>
<p>Livingston s Coefficient kappa ( 2) 297</p>
<p>The Index Sc 297</p>
<p>Outcomes of Using the Single–Test– Administration Reliability Techniques 298</p>
<p>Two–Test–Administration Reliability Techniques 299</p>
<p>Equivalence Reliability 299</p>
<p>Test–Retest Reliability 300</p>
<p>Calculating Reliability for Two–Test Administration Techniques 301</p>
<p>The Phi Coefficient 302</p>
<p>Description of Phi 302</p>
<p>Calculating Phi 302</p>
<p>How High Should Phi Be? 304</p>
<p>The Agreement Coefficient 306</p>
<p>Description of the Agreement Coefficient 306</p>
<p>Calculating the Agreement Coefficient 307</p>
<p>How High Should the Agreement Coefficient Be? 308</p>
<p>The Kappa Coefficient 308</p>
<p>Description of Kappa 308</p>
<p>Calculating the Kappa Coefficient 309</p>
<p>How High Should the Kappa Coefficient Be? 311</p>
<p>Comparison of , 0, and 313</p>
<p>The Logistics of Establishing Test Reliability 314</p>
<p>Choosing Items 314</p>
<p>Sample Test–Takers 315</p>
<p>Testing Conditions 316</p>
<p>Recommendations for Choosing a Reliability Technique 316</p>
<p>Summary Comments 317</p>
<p>15 Reliability of Performance Tests 319</p>
<p>Reliability and Validity of Performance Tests 319</p>
<p>Types of Rating Errors 320</p>
<p>Error of Standards 320</p>
<p>Halo Error 321</p>
<p>Logic Error 321</p>
<p>Similarity Error 321</p>
<p>Central Tendency Error 321</p>
<p>Leniency Error 322</p>
<p>Inter–Rater Reliability 322</p>
<p>Calculating and Interpreting Kappa ( ) 323</p>
<p>Calculating and Interpreting Phi ( ) 335</p>
<p>Repeated Performance and Consecutive Success 344</p>
<p>Procedures for Training Raters 347</p>
<p>What If a Rater Passes Everyone Regardless of Performance? 349</p>
<p>What Should You Do? 352</p>
<p>What If You Get a High Percentage of Agreement Among Raters But a Negative Phi Coefficient? 353</p>
<p>16 Report Scores 357</p>
<p>CRT Versus NRT Reporting 358</p>
<p>Summing Subscores 358</p>
<p>What Should You Report to a Manager? 361</p>
<p>Is There a Legal Reason to Archive the Tests? 362</p>
<p>A Final Thought About Testing and Teaching 362</p>
<p>Part IV: Legal Issues in Criterion–Referenced Testing 365</p>
<p>17 Criterion–Referenced Testing and Employment Selection Laws 367</p>
<p>What Do We Mean by Employment Selection Laws? 368</p>
<p>Who May Bring a Claim? 368</p>
<p>A Short History of the Uniform Guidelines on Employee Selection Procedures 370</p>
<p>Purpose and Scope 371</p>
<p>Legal Challenges to Testing and the Uniform Guidelines 373</p>
<p>Reasonable Reconsideration 376</p>
<p>In Conclusion 376</p>
<p>Balancing CRTs with Employment Discrimination Laws 376</p>
<p>Watch Out for Blanket Exclusions in the Name of Business Necessity 378</p>
<p>Adverse Impact, the Bottom Line, and Affirmative Action 380</p>
<p>Adverse Impact 380</p>
<p>The Bottom Line 383</p>
<p>Affirmative Action 385</p>
<p>Record–Keeping of Adverse Impact and Job–Relatedness of Tests 387</p>
<p>Accommodating Test–Takers with Special Needs 387</p>
<p>Testing, Assessment, and Evaluation for Disabled Candidates 390</p>
<p>Test Validation Criteria: General Guidelines 394</p>
<p>Test Validation: A Step–by–Step Guide 397</p>
<p>1. Obtain Professional Guidance 397</p>
<p>2. Select a Legally Acceptable Validation Strategy for Your Particular Test 397</p>
<p>3. Understand and Employ Standards for Content–Valid Tests 398<br /><br /> 4. Evaluate the Overall Test Circumstances to Assure Equality of Opportunity 399</p>
<p>Keys to Maintaining Effective and Legally Defensible Documentation 400</p>
<p>Why Document? 400</p>
<p>What Is Documentation? 401</p>
<p>Why Is Documentation an Ally in Defending Against Claims? 401</p>
<p>How Is Documentation Used? 402</p>
<p>Compliance Documentation 402</p>
<p>Documentation to Avoid Regulatory Penalties or Lawsuits 404</p>
<p>Use of Documentation in Court 404</p>
<p>Documentation to Refresh Memory 404</p>
<p>Documentation to Attack Credibility 404</p>
<p>Disclosure and Production of Documentation 405</p>
<p>Pay Attention to Document Retention Policies and Protocols 407</p>
<p>Use Effective Word Management in Your Documentation 409</p>
<p>Use Objective Terms to Describe Events and Compliance 412</p>
<p>Avoid Inflammatory and Off–the–Cuff Commentary 412</p>
<p>Develop and Enforce Effective Document Retention Policies 413</p>
<p>Make Sure Your Documentation Is Complete 414</p>
<p>Make Sure Your Documentation Is Capable of "Authentication" 415</p>
<p>In Conclusion 415</p>
<p>Is Your Criterion–Referenced Testing Legally Defensible? A Checklist 416</p>
<p>A Final Thought 419</p>
<p>Epilogue: CRTD as Organizational Transformation 421</p>
<p>References 425</p>
<p>Index 433</p>
<p>About the Authors 453</p>