Made with FlowPaper - Flipbook Maker
Exam Report for 2020-2021 Exam Year Board of Certification (BOC) Certification Exam for Athletic Trainers May 2021 BOC 2020-2021 Exam Year Report Introduction The Board of Certification, Inc., (BOC) is a non-profit credentialing agency that provides certification for the athletic training profession. Although it had already operated for twenty years, the BOC was incorporated in 1989 to govern the certification program, including the exam, for entry-level certification and recertification standards for Athletic Trainers. The entry-level certification program is designed to establish a common benchmark for entry into the athletic training profession. The BOC serves the public interest by developing, administering and continually reviewing a certification process that reflects current standards of practice in athletic training. Standard Setting and Equating of Exam Forms The modified Angoff process was used to establish the performance standard for the exam forms based on the Practice Analysis, 7th Edition (PA7). In April 2017, a panel of 12 Certified Athletic Trainers reviewed the scored items for forms PA701 and PA702. A few minor changes were made to the scored set of items on form PA702. When the forms were introduced in May 2017, the cut score for form PA702 was equated to the standard set on form PA701. All new forms assembled under PA7, including PA707 and PA708, have been equated using item response theory (IRT) methods to the passing standard established for PA701 after the first exam window in which they are used. Score Reporting Because exam forms are possibly of different difficulty, providing raw scores can be misleading. As a result, many programs, including the BOC exam, use scaled scores. Scaled scores are particularly useful because they provide the basis for meaningful long-term comparisons of results across different forms of an exam. Scaled scores are used because over the life of every testing program, situations arise in which changes in test length occur, a decision is made to assess more or fewer areas, the numbers of items that are scored versus unscored (experimental) changes, or forms of the test representing different difficulty are compared. For scaled scores, the passing standard (number of items answered correctly) on the BOC exam form is always reported as the same scaled score. The equated scores are then converted via linear transformation to a scale of 200 to 800 with the passing standard reported as 500. The BOC provides scaled scores and pass/fail decisions to candidates approximately two to four weeks after close of an exam window. Candidates pass or fail based on how their exam performance compares to the equated, criterion-referenced passing standard for the form taken. BOC 2020-2021 Exam Year Report Analysis of the Certification Exam Candidate Performance Candidates in 2020 did not perform at the usual level on the exam. It should be noted that the 2020 testing year coincided with the COVID-19 pandemic, which caused the BOC to cancel the first exam window, normally completed in April each year. It is thought that the pandemic influenced candidate performance because all colleges and universities in the US cancelled in-person instruction, including practical training, and the transition to online instruction was slow and imperfect, meaning that subject matter may not have been taught as effectively as usual. Candidates Excluded from This Report Beginning with 2018, candidates who were administered the BOC certification exam via paper and pencil (i.e., ADA candidates) have been included in the information provided in Table 1. They were not included when the statistics were computed for other years reported in Table 1. These individuals (n = 9) were excluded from the analyses in the rest of the report. Also, candidates who completed less than 25% of their exam were excluded from statistical reports because of the likelihood that their performance was affected by construct irrelevant variance, such as being late to the site, limited English proficiency or other issues. There are 4,953 administrations of the BOC certification exam included in this report, which represents a 10.6% decrease from the 2019-2020 testing year (5,539). The number of first-time exams decreased from 3,913 in 2019- 2020 to 3,035 in 2020-2021, which represents a 28.9% decrease. Retake exams increased from 1,626 in 2019- 2020 to 1,917 in 2020-2021, a 17.9% increase. Pass Rates Table 1. Historical BOC Exam Counts and Pass Rates Year First-time Pass % Pass Retake Pass % Pass All Pass % Pass RD/PA6 2011-2012 3,222 2,653 82.3% 1,664 696 41.8% 4,886 3,269 66.9% 2012-2013 3,631 2,935 80.8% 1,319 507 38.4% 4,950 3,442 69.5% 2013-2014 3,679 3,048 82.8% 1,210 497 41.1% 4,889 3,545 72.5% 2014-2015 3,768 3,039 80.7% 1,283 466 36.3% 5,051 3,505 69.4% 2015-2016 4,059 3,357 82.7% 1,308 547 41.8% 5,367 3,904 72.7% 2016-2017 4,119 3,444 83.6% 1,204 465 38.6% 5,323 3,909 73.4% 2017-2018 2,406 2,149 89.3% 185 85 45.9% 2,591 2,234 86.2% PA7 2017-2018 1,606 1,091 67.9% 1,172 334 28.5% 2,780 1,426 51.3% 2018-2019* 3,974 3,091 77.8% 1,622 645 39.8% 5,596 3,736 66.8% 2019-2020* 3,913 3,038 77.6% 1,626 636 39.1% 5,539 3,674 66.3% 2020-2021* 3,035 1,871 61.6% 1,916 576 30.0% 4,951 2,447 49.4% *Note. Total numbers include ADA and paper and pencil administrations. BOC 2020-2021 Exam Year Report Table 2. Historical BOC Exam Scaled Score Summary Statistics Cohort N Mean Median Std Dev Min Max All 2020-21 4,953 498.1 495 87.7 200 750 First-time 3,035 521.5 520 88.2 200 750 Retake 1,911 461.0 470 72.8 200 690 All 2019-20 5,528 540.2 532 66.1 358 735 First-time 3,912 558.8 558 65.2 382 735 Retake 1,616 495.2 489 42.3 358 683 All 2018-19 5,591 539.0 532 66.3 340 729 First-time 3,970 557.6 555 64.8 352 729 Retake 1,621 493.4 488 44.0 340 642 All 2017-18 5,369 536.9 538 71.3 275 743 First-time 4,012 557.8 560 65.1 304 743 Retake 1,357 475.0 478 49.9 275 665 All 2016-17 5,323 551.3 554 72.2 265 751 First-time 4,119 569.6 577 67.2 315 751 Retake 1,204 488.8 489 50.7 265 678 All 2015-16 5,367 543.5 544 67.5 217 729 First-time 4,059 561.2 564 62.8 356 729 Retake 1,308 488.6 489 49.9 217 661 All 2014-15 5,051 543.3 542 71.4 259 733 First-time 3,768 563.0 567 67.1 272 733 Retake 1,283 485.5 484 48.7 259 650 All 2013-14 4,889 529.7 536 57.1 272 680 First-time 3,679 545.1 548 51.9 332 680 Retake 1,210 482.8 488 45.5 272 650 All 2012-13 4,950 524.0 524 54.0 302 674 First-time 3,631 539.0 548 51.0 302 674 Retake 1,319 484.0 488 41.0 314 632 All 2011-12 4,886 525.0 524 54.0 230 692 First-time 3,222 542.0 548 51.0 272 692 Retake 1,664 491.0 494 44.0 230 644 Exam Form Reliabilities and Other Summary Data Data presented in Table 2 summarize the performance of the forms of the BOC certification exam used during the year. Reliability is assessed using Cronbach’s alpha (Cronbach, 1951), a measure typically used for estimating overall exam form reliability; Brennan-Kane (Brennan & Kane, 1977), a measure used for estimating the decision consistency (i.e., the reliability of pass/fail decisions based on the test); and the standard error of measurement (SEM) presented in raw score units, a measure of the precision of the exam form. BOC 2020-2021 Exam Year Report Summary Statistics concerning the quality of the BOC certification exam as a measurement instrument indicate that the exam complies with psychometric requirements that pertain to certification and licensure tests. Notably, estimates of reliability and equivalence across forms for the various parts of the exam are acceptable. Likewise, candidate performance on all parts of the exam is consistent with the public protection mission of the BOC. References American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). “Standards for Educational and Psychological Testing.” Washington, D.C.: AERA. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. “Journal of Educational Measurement,” 14, 277–289. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. “Psychometrika,” 16, 297–334. Equal Employment Opportunity Commission (EEOC), U.S. Civil Service Commission, U.S. Department of Labor, and U.S. Department of Justice. (1978). Uniform Guidelines on Employee Selection Procedures. “Federal Register,” 43 (166), 38290–38315. Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. “Journal of Educational Measurement”, 34, 353–366. National Commission for Certifying Agencies (2016). “Standards for the Accreditation of Certification Programs.” Washington, DC: Institute for Credentialing Excellence. Kolen, M. J., & Brennan, R. L. (2004) Test “Equating, Scaling and Linking: Methods and Practices Statistics for Social Science and Behavioral Sciences” (2 ed.). Springer-Verlag New York Inc. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability.”Psychometrika,” 2, 151–160. Next >