Modern Psychometrics

1. The history and evolution of psychometric testing

Introduction

What is psychometrics?

Psychometrics in the 21st century

History of assessment

1. 1. 
      
      Chinese origins
      
      The ability to learn
      
      The nineteenth century

Beginnings of psychometrics as a science

1. 1. 
      
      Intelligence testing
      
      Eugenics and the dark decades

Psychometric testing of ability

1. 1. The dark ages come to an end
      
      An abundance of abilities

Tests of other psychological constructs

1. 1. 
      
      Personality
      
      Integrity
      
      Interests
      
      Motivation
      
      Values
      
      Temperament
      
      Attitude
      
      Belief

Summary

2. Constructing your own psychometric questionnaire

The purpose of the questionnaire

Making a blueprint

Writing items

1. 1. Alternate-choice items
      
      Multiple-choice items
      
      Rating-scale items
      
      All questionnaires
      
      Knowledge-based questionnaires
      
      Person-based questionnaires

Designing the questionnaire

Piloting the questionnaire

Item analysis

1. 1. 
      
      Facility
      
      Discrimination
      
      Distractors

Obtaining the reliability

1. 1. 
      
      Cronbach’s alpha
      
      Split-half reliability

Assessing validity

1. 1. Face validity
      
      Content validity

Standardization

3. The Psychometric principles

Reliability

1. 1. Test-retest reliability
      
      Parallel-forms reliability
      
      Split-half reliability
      
      Interrater reliability
      
      Internal consistency
      
      The standard error of measurement (SEM)
      
      Comparing test reliabilities
      
      Restriction of range

Validity

1. 1. Face validity
      
      Content validity
      
      Predictive validity
      
      Concurrent validity
      
      Construct validity
      
      Differential validity

Standardization

1. 1. Norm referencing
      
      Criterion referencing

Equivalence

1. 1. Differential item functioning
      
      Measurement invariance
      
      Adverse impact

Summary

4. Psychometric measurement

True-score theory

Identification of latent traits with factor analysis

1. 1. Spearman’s two-factor theory
      
      Vector algebra and factor rotation
      
      Moving into more dimensions
      
      Multidimensional scaling

Application of factor analysis to test construction

1. 1. Eigenvalues
      
      Identifying the number of factors to extract using the Kaiser criterion
      
      Identifying the number of factors to extract using the Cattell scree test
      
      Other techniques for identifying the number of factors to extract
      
      Factor rotation
      
      Rotation to simple structure
      
      Orthogonal rotation
      
      Oblique rotation

Limitations of the classical factor-analystic approach

Criticisms of psychometric measurement theory

1. 1. The Platonic true score
      
      Psychological vs. physical true scores
      
      Functional assessment and competency testing
      
      Machine learning and the black box

Summary

5. Item response theory and computer adaptive testing

Introduction

Item banks

1. 1. 
      
      The Rasch model
      
      Assessment of educational standards
      
      The Birnbaum model

The evolution of modern psychometrics

1. 1. 
      
      Computer adaptive testing
      
      Item equating
      
      Polytomous IRT

An intuitive graphical description of item tesponse theory

1. 1. 
      
      Limitations of classical test theory

A graphical Introduction to item response theory

1. 1. 
      
      The logistic curve
      
      3PL-model: difficulty parameter
      
      3PL model: discrimination parameter
      
      3PL model: guessing parameter
      
      The Fisher information function
      
      The test information function and its relationship to the standard error of measurement
      
      How to score an IRT test

Principles of computer adaptive testing

Summary of item response theory

Confirmatory factor analysis

6. Personality theory

Theories of personality

1. 1. Psychoanalytic theory
      
      Humanistic theory
      
      Social learning theory
      
      Behavioral genetics
      
      Type and trait theories
      
      Different approaches to personality assessment

1. 1. 
      
      Self-report techniques and personality profiles
      
      Reports by others
      
      Online digital footprints
      
      Situational assessments
      
      Projective measures
      
      Observations of behavior
      
      Task performance methods
      
      Polygraph methods
      
      Repertory grids

Sources and management of bias

1. 1. 
      
      Self-report techniques and personality profiles
      
      Reports by others
      
      Online digital footprints
      
      Situational assessments
      
      Projective measures
      
      Observations of behavior
      
      Task performance methods
      
      Polygraph methods
      
      Repertory grids

Informal methods of personality assessment

State versus trait measures

Ipsative scaling

Spurious validity and the Barnum Effect

Summary

7. Personality assessment in the workplace

Prediction of successful employment outcomes

1. 1. Validation of personality questionnaires previously used in employment
      
      Historical antecedents to the five-factor model
      
      Stability of the five-factor model
      
      Cross-cultural aspects of the five-factor model
      
      Scale independence and the role of facets
      
      Challenges to scale construction for the five-factor model
      
      Impression management
      
      Acquiescence
      
      Response bias and factor structure
      
      Development of the five OBPI personality scales

Assessing counterproductive behavior at work

1. 1. The impact of behaviorism
      
      Prepsychological theories of integrity
      
      Modern integrity testing
      
      Psychiatry and the medical model
      
      The dysfunctional tendencies
      
      The dark triad
      
      Assessing integrity at work
      
      The OBPI integrity scales

Conclusion

8. Employing digital footprints in psychometrics

Introduction

Types of digital footprint

1. 1. 
      
      Usage logs
      
      Language data
      
      Mobile sensors
      
      Images and audiovisual data

Typical applications of digital footprints in psychometrics

1. 1. 
      
      Replacing and complimenting traditional measures
      
      New contexts and new constructs
      
      Predicting future behavior
      
      Studying human behavior
      
      Supporting the development of traditional measures

Advantages and challenges of employing digital footprints in psychometrics

1. 1. 
      
      High ecological validity
      
      Greater detail and longitude
      
      Less control over the assessment environment
      
      Greater speed and unobtrusiveness
      
      Less privacy and control
      
      No anonymity
      
      Bias
      
      Enrichment of existing constructs
      
      Developing digital-footprint-based psychometric measures
      Collecting digital footprints

1. 1. How much data is needed?

Preparing digital footprints for analysis

1. 1. 
      
      Respondent-footprint matrix
      
      Data sparsity

Reducing the dimensionality of the respondent-footprint matrix

1. 1. 
      
      Singular value decomposition
      
      Latent Dirichlet allocation

Building prediction models

9. Psychometrics in the era of the intelligent machine

History of computerization in psychometrics

1. 1. 
      
      Computerized statistics
      
      Computerized item banks
      
      Computerized item generation
      
      Automated advice and report systems

The evolution of AI in psychometrics

1. 1. 
      
      Expert systems
      
      Neural networks (machine learning)
      
      Parallel processing
      
      Predicting with statistics and machine learning
      
      Explainability

Psychometrics in cyberspace

1. 1. 
      
      What and where is cyberspace?
      
      The medium is the message

Moral development in AI

1. 1. 
      
      Kohlberg’s theory of moral development
      
      Do machines have morals?
      
      The laws of robotics
      
      Artificial general intelligence

Conclusion