Made with FlowPaper - Flipbook Maker
Exam Report for 2023 Testing Year Board of Certification (BOC) Orthopedic Specialty Exam May 2024 2 of 5 BOC 2023 Exam Year Report Introduction The Board of Certification, Inc., (BOC) is a non-profit credentialing agency that provides certification for the athletic training profession. Although it had already operated for 20 years as a committee of the National Athletic Trainers Association, the BOC was incorporated in 1989 to govern the certification program for entry-level certification for Athletic Trainers (AT) and the standards for recertification. The BOC serves the public interest by developing, administering and continually reviewing a certification process that reflects current standards of practice in athletic training. In 2021, the BOC Orthopedic Specialty Certification (orthopedic specialty) for ATs was launched. The Board-Certified Specialist in Orthopedics (BCS-O) credential is available to ATs who have acquired specialized education and focused experience in orthopedics beyond the requirements for the ATC® credential. It is the first and only board-certified specialty for ATs who specialize in orthopedics, and it is backed by the BOC’s rigorous process and standards. Standard Setting and Equating of Exam Forms The modified Angoff process was used to establish the performance standard for the Orthopedic Specialty Exam (OSE). In October 2021, a panel of eight certified ATs with a specialty practice in orthopedics participated in the study. All new forms will be equated to the passing standard established for Form PA101 using a method that is appropriate for small programs after the first exam window in which they are used. Score Reporting Raw scores for the OSE are reported as scaled scores. Scaled scores are particularly useful because they provide the basis for meaningful long-term comparisons of results across different forms of an exam. Scaled scores are used because, over the life of every exam program, situations arise in which changes in exam length occur, a decision is made to assess more or fewer areas, and the number of items that are scored versus unscored (field test) changes or forms of the exam of different difficulty are compared. The equated scores are then converted via linear transformation to a scale of 1000 to 1450 with the passing standard reported as 1200. The BOC provides scaled scores and pass/fail standing to applicants approximately two to four weeks after the close of an exam window. Applicants pass or fail based on how their exam performance compares to the criterion-referenced passing standard. Because the content areas specified for the exam were validated as critical for specialty practice, the items and the forms are intended to assess critical knowledge and/or skill for newly certified specialists. Overall scores from the exam can be used to make inferences about the relevant knowledge and skills that applicants have acquired. Scores on the OSE are not, however, intended as predictors of future success in the specialty. 3 of 5 BOC 2023 Exam Year Report Analysis of the OSE Applicant Performance Applicants may qualify to take the OSE by completing a residency in orthopedics or by satisfying alternative eligibility criteria. Also, applicants taking the OSE in the period addressed in this report were classified as either first-time exams or retake exams: ▪ First-time exams – exams taken by applicants who never previously sat for any form of the OSE ▪ Retake exams – exams taken by applicants who previously sat one or more times for OSE There are twenty-two exam events of the OSE included in this report over two administration windows, which represents a 29.4% increase from the 2022 testing year (17). Of the 22 exam events, five (22.7%) were graduates of an athletic training residency in orthopedics and 17 (77.3%) qualified by meeting other requirements. Also, 15 (68.2%) of the exam events were applicants taking the exam for the first time, and seven (31.8%) were retake attempts, each a unique individual (see Table 2). Pass Rates Table 1 provides annual pass rates for the OSE exam since 2021 by eligibility pathway; and Table 2 provides annual pass rate by Retake status. Table 1. Historical Counts and Pass Rates for the OSE Overall and by Eligibility Pathway Year Path 1 Residency Pass % Pass Path 2 Other Pass % Pass All Pass % Pass PA103 2021 5 5 100.0% 10 4 40.0% 15 9 60.0% 2022 2 1 50.0% 15 6 40.0% 17 7 41.2% 2023 5 3 60.00% 17 8 47.06% 22 11 50.00% Table 2. Historical Counts and Pass Rates for the OSE Overall and by Retake Status Year First-time Pass % Pass Retake Pass % Pass All Pass % Pass PA103 2021 15 9 60.0% --- --- --- 15 9 60.0% 2022 12 5 41.7% 5 2 40.0% 17 7 41.2% 2023 15 9 60.00% 7 2 28.57% 22 11 50.00% 4 of 5 BOC 2023 Exam Year Report Distribution of Candidate Scores Table 3 presents scaled score summary statistics for the 2023 annual testing cohort, alongside the previous two years of testing. Appendix A contains definitions and interpretational information for the statistics contained in the report. Table 3. Historical OSE Scaled Score Summary Statistics Grouped by Retake Status Cohort N Mean Median Std Dev Min Max All 2023 22 1196.2 1203 42.9 1122 1272 First-time 15 1205.6 1218 43.7 1134 1272 Retake 7 1176.0 1182 35.7 1122 1212 All 2022 17 1196.1 1188 65.7 1110 1368 First-time 12 1201.5 1188 74.6 1110 1368 Retake 5 1183.2 1176 41.0 1134 1236 All 2021 15 1233.2 1200 97.3 1104 1410 First-time 15 1233.2 1200 97.3 1104 1410 Retake -- -- -- -- -- -- Exam Form Reliabilities and Other Summary Data Data presented in Table 4 summarizes the performance of the forms of the BOC certification exam used during the annual administration windows. Reliability is assessed using Cronbach’s alpha (Cronbach, 1951), the most widely used measure of overall exam form reliability; Livingston-Lewis (Livingston & Lewis, 1995), a measure used for estimating the decision consistency (i.e., the reliability of pass/fail decisions based on the test); and the standard error of measurement (SEM) presented in raw score units, a measure of the precision of the exam form. Reliability and decision consistency estimates for PA101 were all in the acceptable range, with the exception of the February 2023 examination. The February administration was the exact set of scored items administered in the three prior administrations which all achieved acceptable reliability criteria. The newest form, PA102, was administered in September 2023 and yielded acceptable reliability measures. Summary and Conclusions Statistics concerning the quality of the OSE as a measurement instrument indicate that the exam complies with psychometric requirements that pertain to certification and licensure exams. Notably, estimates of reliability across forms of the exam are acceptable. Likewise, applicant performance on all parts of the exam is consistent with the public protection mission of the BOC. 5 of 5 BOC 2023 Exam Year Report References American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, D.C.: AERA. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277–289. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. Equal Employment Opportunity Commission (EEOC), U.S. Civil Service Commission, U.S. Department of Labor, and U.S. Department of Justice. (1978). Uniform Guidelines on Employee Selection Procedures. Federal Register, 43 (166), 38290–38315. Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353–366. National Commission for Certifying Agencies (2021). Standards for the Accreditation of Certification Programs. Washington, DC: Institute for Credentialing Excellence. Kolen, M. J., & Brennan, R. L. (2004) Test Equating, Scaling and Linking: Methods and Practices Statistics for Social Science and Behavioral Sciences (2 ed.). Springer-Verlag New York Inc. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151–160. Next >