Made with FlowPaper - Flipbook Maker
Exam Report for 2021 and 2022 Board of Certification (BOC) Orthopedic Specialty Exam March 2023 Credentialing Examination Consulting, LLC 2 of 4 BOC 2021-2022 Exam Year Report Introduction The Board of Certification, Inc., (BOC) is a non-profit credentialing agency that provides certification for the athletic training profession. Although it had already operated for 20 years as a committee of the National Athletic Trainers Association, the BOC was incorporated in 1989 to govern the certification program for entry-level certification for Athletic Trainers (AT) and the standards for recertification. The BOC serves the public interest by developing, administering and continually reviewing a certification process that reflects current standards of practice in athletic training. In 2021, the BOC Orthopedic Specialty Certification (orthopedic specialty) for ATs was launched. The Board Certified Specialist in Orthopedics (BCS-O) credential is available to ATs who have acquired specialized education and focused experience in orthopedics beyond the requirements for the ATC® credential. It is the first and only board-certified specialty for ATs who specialize in orthopedics, and it is backed by the BOC’s rigorous process and standards. Standard Setting and Equating of Exam Forms The modified Angoff process was used to establish the performance standard for the Orthopedic Specialty Exam (OSE). In October 2021, a panel of eight certified ATs with a specialty practice in orthopedics participated in the study. All new forms will be equated to the passing standard established for Form PA101 using a method that is appropriate for small programs after the first exam window in which they are used. Score Reporting Raw scores for the OSE are reported as scaled scores. Scaled scores are particularly useful because they provide the basis for meaningful long-term comparisons of results across different forms of an exam. Scaled scores are used because over the life of every exam program, situations arise in which changes in exam length occur, a decision is made to assess more or fewer areas, the number of items that are scored versus unscored (field test) changes or forms of the exam of different difficulty are compared. The equated scores are then converted via linear transformation to a scale of 1000 to 1450 with the passing standard reported as 1200. The BOC provides scaled scores and pass/fail standing to applicants approximately two to four weeks after the close of an exam window. Applicants pass or fail based on how their exam performance compares to the criterion-referenced passing standard. Because the content areas specified for the exam were validated as critical for specialty practice, the items and the forms are intended to assess critical knowledge and/or skill for newly certified specialists. Overall scores from the exam can be used to make inferences about the relevant knowledge and skills that applicants have acquired. Scores on the OSE are not, however, intended as predictors of future success in the specialty. 3 of 4 BOC 2021-2022 Exam Year Report Analysis of the OSE Applicant Performance Applicants may qualify to take the OSE by completing a residency in orthopedics or by satisfying alternative eligibility criteria. Also, applicants taking the OSE in the period addressed in this report were classified as either first-time exams or retake exams: ▪ First-time exams – exams taken by applicants who never previously sat for any form of the OSE ▪ Retake exams – exams taken by applicants who previously sat one or more times for OSE Thirty-two exam events of the OSE over three exam windows are included in this report, which represents all of the exam events that have taken place to date (2021-2022). Of the 32 exam events, seven (21.9%) were graduates of an athletic training residency in orthopedics and 25 (78.1%) qualified by meeting other requirements. Also, 27 (84.4%) of the exam events were applicants taking the exam for first time, and five (15.6%) were retake attempts by four individuals. Of the 32 exam events, 15 were in the inaugural exam window (Oct. 2021), for which all were first-time attempts. Pass Rates Table 1. Counts and Pass Rates for the OSE Overall and by Eligibility Pathway Year Path 1 Residency Pass % Pass Path 2 Other Pass % Pass All Pass % Pass PA103 2021 5 5 100.0% 10 4 40.0% 15 9 60.0% 2022 2 1 50.0% 15 6 40.0% 17 7 41.2% Table 2. Counts and Pass Rates for the OSE Overall and by Retake Status Year First-time Pass % Pass Retake Pass % Pass All Pass % Pass PA103 2021 15 9 60.0% --- --- --- 15 9 60.0% 2022 12 5 41.7% 5 2 40.0% 17 7 41.2% Table 3. Historical OSE Scaled Score Summary Statistics Grouped by Retake Status Cohort N Mean Median Std Dev Min Max All 2022 17 1196.1 1188 65.7 1110 1368 First-time 12 1201.5 1188 74.6 1110 1368 Retake 5 1183.2 1176 41.0 1134 1236 All 2021 15 1233.2 1200 97.3 1104 1410 First-time 15 1233.2 1200 97.3 1104 1410 Retake -- -- -- -- -- -- 4 of 4 BOC 2021-2022 Exam Year Report Exam Form Reliabilities and Other Summary Data The performance of the forms of the OSE used during 2021-2022 is consistent with reporting requirements for the National Commission for Certifying Agencies accreditation. Reliability is assessed using Cronbach’s alpha (Cronbach, 1951), a measure typically used for estimating overall exam form reliability; Brennan-Kane (Brennan & Kane, 1977), a measure used for estimating the decision consistency (i.e., the reliability of pass/fail decisions based on the exam; and the standard error of measurement (SEM) presented in raw score units, a measure of the precision of the exam form. Summary and Conclusions Statistics concerning the quality of the OSE as a measurement instrument indicate that the exam complies with psychometric requirements that pertain to certification and licensure exams. Notably, estimates of reliability across forms of the exam are acceptable. Likewise, applicant performance on all parts of the exam is consistent with the public protection mission of the BOC. References American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, D.C.: AERA. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277–289. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. Equal Employment Opportunity Commission (EEOC), U.S. Civil Service Commission, U.S. Department of Labor, and U.S. Department of Justice. (1978). Uniform Guidelines on Employee Selection Procedures. Federal Register, 43 (166), 38290–38315. Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353–366. National Commission for Certifying Agencies (2021). Standards for the Accreditation of Certification Programs. Washington, DC: Institute for Credentialing Excellence. Kolen, M. J., & Brennan, R. L. (2004) Test Equating, Scaling and Linking: Methods and Practices Statistics for Social Science and Behavioral Sciences (2 ed.). Springer-Verlag New York Inc. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151–160. Next >