What’s in a Keystroke? Working toward an Automated Process Evaluation Using Paradata

Catherine N. Barry, University of California, Berkeley
Rachael Walsh, U.S. Census Bureau

Survey instruments are data collection tools that are improved and re-designed based on testing and feedback from field representatives, respondents, and survey designers. Assessing paradata is another useful method for survey instrument evaluation. Paradata are ancillary information generated throughout the data collection process, including data such as keystrokes and timestamps. Using the 2011 Survey of Income and Program Participation-Event History Calendar (SIPP-EHC), this study explores how keystrokes used by interviewers within the SIPP-EHC are associated with interviewer and survey characteristics. Results suggest that more experienced interviewers have successful strategies for avoiding ‘don’t know’ item non-response and that survey Help Screens should be tailored to non-response situations when interviewers are unable to match a response with a question’s available options. Overall, results highlight the evaluation of paradata for enhancing data quality, interviewer know-how and the survey instrument by gaining a better understanding of how interviewers and respondents interact with the instrument.