Published in: Large-scale Assessments in Education

Abstract:

Background

About half of the countries participating in TIMSS 2019 transitioned from paper-based to digital assessment in that cycle, with most remaining countries completing the transition in TIMSS 2023. All countries transitioning in 2019 completed a paper-based ‘bridge’ study to examine mode effects (differences in achievement related to the mode of administration) and ensure the comparability of scores from previous cycles. However, only four countries opted to administer a national comparison (bridge) study in 2023: Ireland, Azerbaijan, Bahrain, and Flanders (Belgium). This Brief Report documents findings from these four national comparison studies. The aim is to verify the extent to which the transition to digital testing may have affected student performance in these countries, noting that ten countries in TIMSS 2019 were previously found to have displayed country-level mode effects in at least one subject or grade.

Findings

We find no evidence of significant differences in overall performance by mode of administration, in either mathematics or science, in any of the four included countries. We also find no evidence of significant mode effects among boys or girls in Ireland, Flanders, or Bahrain. In Azerbaijan, boys achieved a significantly higher science score on the digital version of TIMSS compared to the paper version. Some differences in achievement by mode are observed at the level of the mathematics and science content and cognitive domains. These are limited and present no distinctive pattern in three of the countries studied although, again, more consistent differences are noted in Azerbaijan. Finally, some preliminary evidence is presented of differences in student performance on the digital assessment related to students’ access to a computer at home in Flanders, Bahrain, and Azerbaijan.

Conclusions

The findings provided here confirm that achievement estimates in TIMSS 2023 in these four countries were substantively unaffected at a high level by the change in the mode of administration. This should strengthen confidence in the use of these data for policymaking, educational practice, and further research. In particular, trend data from 2019 to 2023 in these countries can be more confidently interpreted as reflecting substantive trends or changes in achievement, giving researchers greater confidence that trends have not been significantly confounded by the procedural change of assessment mode. Evidence of some specific differences in each country could be considered further by national experts in curriculum and assessment. In particular, several differences have been observed that merit further consideration by researchers and policymakers in Azerbaijan.

Citation: Clerkin, A., Verhelst, D., Mahdi, A., Mammadov, S & McHugh, G. (2026). Mode effects in the transition to digital testing: Evidence from TIMSS in Ireland, Azerbaijan, Bahrain, and Flanders. Large-scale Assessments in Education, 14, 4. https://doi.org/10.1186/s40536-026-00313-x