Made with FlowPaper - Flipbook Maker
2023-2024 Exam Report Certification Exam for Athletic Trainers Prepared by: Ian Hembry, PhD Director of Psychometrics and Examinations May 2024 BOC 2023-2024 Testing Year Report 1 Introduction The Board of Certification, Inc., (BOC) is a non-profit credentialing agency that provides certification for the athletic training profession. Although it had already operated for 20 years as a committee of the National Athletic Trainers Association, the BOC was incorporated in 1989 to govern the certification program and the standards for recertification. The entry-level certification exam is designed to establish a common benchmark for the profession. The BOC serves the public interest by developing, administering and continually reviewing a certification process that reflects current standards of practice in athletic training. Standard Setting and Equating of Exam Forms The modified Angoff process was used to establish the performance standard for the forms based on PA8. In April 2023, a panel of nine certified ATs reviewed the scored items for forms PA801 and PA802. The cut score for Form PA802 was equated to the standard set on Form PA801. All new forms assembled under PA8, including PA802, will be equated using the Rasch model. The logit equivalent will establish for the new form in reference to the standard established for PA801 after the first testing window in which they appear. Score Reporting Because exam forms are possibly of different difficulty, providing raw scores can be misleading. As a result, many programs, including the BOC exam, use scaled scores. Scaled scores are particularly useful because they provide the basis for meaningful long-term comparisons of results across different forms of an exam. Scaled scores are used because, over the life of every testing program, situations arise in which changes in test length occur, a decision is made to assess more or fewer areas, the number of items that are scored versus unscored (experimental) changes or forms of the test of different difficulty are compared. To ensure the scores have comparable meaning as these changes occur, the equated scores developed for all candidates are converted via linear transformation to a scale of 200 to 800 with the passing standard reported as 500. The BOC provides scaled scores and pass/fail standing to candidates approximately two to four weeks after the close of a testing window. Candidates pass or fail based on how their test performance compares to the equated, criterion- referenced passing standard for the form taken. Analysis of the Certification Exam Candidate Performance Statistical reports are generated for each form and testing window. All BOC exams administered during the 2023- 2024 testing year have been classified as either first-time exams or retake exams: ▪ First-time exams – exams taken by candidates who never previously sat for any form of the BOC exam ▪ Retake exams – exams taken by candidates who previously sat one or more times for the BOC exam Candidates Excluded from This Report Beginning in 2018, candidates who were administered the BOC certification exam via paper and pencil (i.e., ADA candidates) have been included in the information provided in Table 1. They were not included when the statistics were computed for other years reported in Table 1. Also, for the testing year 2023-2024, these candidates (n = 6 individuals, but 10 testing events) are excluded from the analyses in the rest of the report. No candidates completed less than 25% of their exam, but if there had been they would have been excluded from statistical reports because of the likelihood that their performance was affected by construct irrelevant variance, such as being late to the site, limited English proficiency or other issues. There are 3,876 administrations of the BOC certification exam included in this report, which represents a 3.5% increase from the 2022-2023 testing year (3,744). The number of first-time exams decreased from 2,427 in 2022- 2023 to 2,288 in 2023-2024, which represents a 5.8% decrease. Retake exams increased from 1,317 in 2022- 2023 to 1,591 in 2023-2024, a 20.8% increase. BOC 2023-2024 Testing Year Report 2 Pass Rates Table 1. Historical BOC Exam Counts and Pass Rates Year First-time Pass % Pass Retake Pass % Pass All Pass % Pass PA7 2017-2018 1,606 1,091 67.9% 1,172 334 28.5% 2,780 1,426 51.3% 2018-2019* 3,974 3,091 77.8% 1,622 645 39.8% 5,596 3,736 66.8% 2019-2020* 3,913 3,038 77.6% 1,626 636 39.1% 5,539 3,674 66.3% 2020-2021* 3,035 1,871 61.6% 1,916 576 30.0% 4,951 2,447 49.4% 2021-2022* 2,625 2,027 77.2% 1,495 611 40.9% 4,120 2,638 64.0% 2022-2023* 2,427 1,799 74.1% 1,317 477 36.2% 3,744 2,276 60.8% PA8 2023-2024* 2,285 1569 68.7% 1,591 463 29.1% 3,876 2,032 52.4% *Note. Total numbers include ADA paper and pencil administrations. Table 2. Historical BOC Exam Scaled Score Summary Statistics Cohort N Mean Median Std Dev Min Max All 2023-24 3,866 502.0 500 100.4 200 800 First-time 2,282 537.1 540 98.9 200 800 Retake 1,584 451.4 450 78.7 200 750 All 2022-23 3,738 525.0 530 100.0 200 800 First-time 2,424 556.6 570 96.8 200 800 Retake 1,314 466.8 470 77.1 200 710 All 2021-22 4,118 532.6 540 107.7 200 800 First-time 2,624 568.4 580 102.9 200 800 Retake 1,494 469.8 480 84.8 200 710 All 2020-21 4,953 498.1 495 87.7 200 750 First-time 3,035 521.5 520 88.2 200 750 Retake 1,911 461.0 470 72.8 200 690 All 2019-20 5,528 540.2 532 66.1 358 735 First-time 3,912 558.8 558 65.2 382 735 Retake 1,616 495.2 489 42.3 358 683 All 2018-19 5,591 539.0 532 66.3 340 729 First-time 3,970 557.6 555 64.8 352 729 Retake 1,621 493.4 488 44.0 340 642 All 2017-18 5,369 536.9 538 71.3 275 743 First-time 4,012 557.8 560 65.1 304 743 Retake 1,357 475.0 478 49.9 275 665 BOC 2023-2024 Testing Year Report 3 Exam Form Reliabilities and Other Summary Data The performance of the forms of the BOC certification exam used during the year is consistent with reporting requirements for the National Commission for Certifying Agencies accreditation. Reliability is assessed using Cronbach’s alpha (Cronbach, 1951), the most widely used measure of overall exam form reliability; Livingston- Lewis (Livingston & Lewis, 1995), a measure used for estimating the decision consistency (i.e., the reliability of pass/fail decisions based on the test); and the standard error of measurement (SEM) presented in raw score units, a measure of the precision of the exam form. The results indicate that each scored set meets general guidelines for reliabilities. In addition, the standard errors of measurement (SEM) are acceptably small, which is an indication that each scored set is functioning as an acceptably precise measurement instrument. Summary Statistics concerning the quality of the BOC certification exam as a measurement instrument indicate that the exam complies with psychometric requirements that pertain to certification and licensure tests. Reliability coefficients were consistent with prior testing years. Estimates of equivalence across forms for the various parts of the exam are acceptable. Likewise, candidate performance on all parts of the exam is consistent with the public protection mission of the BOC. References American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, D.C.: AERA. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. Equal Employment Opportunity Commission (EEOC), U.S. Civil Service Commission, U.S. Department of Labor, and U.S. Department of Justice. (1978). Uniform Guidelines on Employee Selection Procedures. Federal Register, 43 (166), 38290–38315. Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353–366. Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on scores. Journal of Educational Measurement, 32, 179-197. National Commission for Certifying Agencies (2016). Standards for the Accreditation of Certification Programs. Washington, DC: Institute for Credentialing Excellence. Kolen, M. J., & Brennan, R. L. (2004) Test Equating, Scaling and Linking: Methods and Practices Statistics for Social Science and Behavioral Sciences (2 ed.). Springer-Verlag New York Inc. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151– 160. Next >