[Home ] [Archive]   [ فارسی ]  
:: Main :: About :: Current Issue :: Archive :: Search :: Submit :: Contact ::
Main Menu
Home::
Journal Information::
Articles archive::
For Authors::
For Reviewers::
Registration::
Contact us::
Site Facilities::
::
Search in website

Advanced Search
..
Receive site information
Enter your Email in the following box to receive the site news and information.
..
Registered in

AWT IMAGE

AWT IMAGE

..
Open Access Policy
..
:: Volume 11, Issue 3 (2019) ::
RME 2019, 11(3): 59-70 Back to browse issues page
The Use of Rasch and Item Mapping in Determining Cut Score of Comprehensive Pre internship Exam
Sh Habibzadeh , A Delavar * , NA Farrokhi , A Minaei , M Jalili
Department of Assessment & Measurment, Faculty of psychology & Education, Allame tabatabaei University, Tehran, Iran , delavarali@yahoo.com
Abstract:   (3807 Views)
Introduction: One of the most controversial issues in educational measurement is to determine the cut score during the development and administration of a test. This paper aimed at using the Rasch model (item mapping) in determining cut score of the comprehensive pre internship exam in medical education.
Methods:This is a descriptive cross-sectional study. in doing so 13 faculty members of Tehran University of Medical Sciences were selected as the panelists Data collection instrument was the 200-item pre intership exam which administered in the Region 10 of the national medical science system between August 22 and September 21, 2016, that  were responded by 274 students of university of medical sciences A cut score was set to identify pass/fail score for performance assessment. Data were analyzed by Winstep.
Result: According to the findings, the item reliability of this exam was 0.98 (PR=0.98) and person reliability of 0.93 (IP=0.93), that indicate the sample variance and test length were suitable and the items were selected from the nine regions proportionally. Examining items by panelists using the item map method showed three judgment stages included the item difficulty of (-0.46), raw score of 103, and ability parameter equivalent of (0.58).
Conclusion: Procedural validity of the cut score showed that the use of item map method resulted in greater agreement between the panelists on this score. Results indicated high satisfaction of the panelists because of feasibility of this process and implementation stage.
Keywords: Cut Score, Rosch Model, Item Mapping, comprehensive preinternship, exam
Full-Text [PDF 605 kb]   (955 Downloads)    
Type of Study: Research | Subject: assessment and evaluation
References
1. Epstein RM, Hundert EM. Defining and assessing professional competence. Jama 2002 9: 287 (2): 226- 35. [DOI:10.1001/jama.287.2.226]
2. Kane MT. Examinee- centered vs. task-centered standard setting. In the Proceedings of the Joint Conference on Standard Setting for Large-scale Assessment 1995; 2, 119-141.
3. Schnabel SD. A Comparison of the Angoff and Item Mapping Standard Setting Methods for a Certification Examination [dissertation].Illinois University: 2018.
4. Cizek GJ, Setting performance standards: Foundations, methods, and innovations. Routledge; 2012. [DOI:10.4324/9780203848203]
5. Çetin S, Gelbal S. A Comparison of Bookmark and Angoff Standard Setting Methods. Educational Sciences: Theory and Practice 2013: 13 (4): 2169- 75.
6. Karantonis A, Sireci SG. The bookmark standard‐ setting method: A literature review. Educational Measurement: Issues and Practice 2006: 25 (1): 4- 12. [DOI:10.1111/j.1745-3992.2006.00047.x]
7. -Nasri Kh, Kahbazi M, Nasri Sh. [Medical Students' Viewpoints toward Basic Sciences and Preinternship Comprehensive Exams in Arak University of Medical Sciences]. Iranian J Med Educ 2010: 10 (1): 82- 91. [Persian]
8. Wang N. Use of the Rasch IRT model in standard setting: An item‐ mapping method. Journal of Educational Measurement. 2003 Sep; 40 (3): 231- 53. [DOI:10.1111/j.1745-3984.2003.tb01106.x]
9. Jaeger RM, Mullis IV, Bourque ML, Shakrani S. Setting performance standards for performance assessments: Some fundamental issues, current practice, and technical dilemmas. Technical issues in large-scale performance assessment 1996: 79- 115.
10. Jiao H, Lissitz RW, Macready G, Wang S, Liang S. Exploring levels of performance using the mixture Rasch model for standard setting1. Psychological Test and Assessment Modeling 2011 1; 53 (4): 499.
11. Yim M. Comparison of results between modified- Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination. Korean journal of medical education 2018: 30 (4): 347. [DOI:10.3946/kjme.2018.110]
12. Hambleton RK. Setting performance standards on educational assessments and criteria for evaluating the process. In Setting performance standards. Routledge; 2013.
13. Kaftandjieva F. Methods for setting cut scores in criterion-referenced achievement tests. Cito, Arnhem: EALTA; 2010.
14. Violato C, Marini A, Lee C. A validity study of expert judgment procedures for setting cutoff scores on high-stakes credentialing examinations using cluster analysis. Evaluation & the health professions 2003; 26 (1): 59- 72. [DOI:10.1177/0163278702250082]
15. Bin Khairani AZ, bin Abd Razak N, Shamsuddin H. Application of the Rasch model and the bookmark method in setting cut scores in mathematics. International Journal of Information and Education Technology 2014; 4 (2): 198. [DOI:10.7763/IJIET.2014.V4.398]
16. Hambleton RK, Swaminathan H, Rogers HJ. Fundamentals of item response theory. Sage; 1991.
17. Näsström G, Nyström P. A comparison of two different methods for setting performance standards for a test with constructed- response items. Practical Assessment, Research & Evaluation 2008; 13 (9):1- 12.
18. Cizek GJ, Bunch MB. Standard setting: A guide to establishing and evaluating performance standards on tests. SAGE Publications Ltd; 2007.
19. Beuk CH. A method for reaching a compromise between absolute and relative standards in examinations. Journal of Educational Measurement 1984 ; 21 (2): 147- 52. [DOI:10.1111/j.1745-3984.1984.tb00226.x]
20. Buckendahl CW, Smith RW, Impara JC, Plake BS. A comparison of Angoff and Bookmark standard setting methods. Journal of Educational measurement: 39 (3): 253- 63. [DOI:10.1111/j.1745-3984.2002.tb01177.x]
21. Lee G, Lewis DM. A generalizability theory approach to standard error estimates for bookmark standard settings. Educational and Psychological Measurement 2008: 68 (4): 603- 20. [DOI:10.1177/0013164407312603]
22. Cizek GJ. Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates; 2001.
23. Lypson ML, Downing SM, Gruppen LD, Yudkowsky R. Applying the Bookmark method to medical education: Standard setting for an aseptic technique station. Medical teacher 2013; 35 (7): 581- 5. [DOI:10.3109/0142159X.2013.778395]
24. Plake BS. Setting performance standards for professional licensure and certification. Applied Measurement in Education 1998 1; 11 (1): 65- 80. [DOI:10.1207/s15324818ame1101_4]
25. Chang L. Judgmental item analysis of the Nedelsky and Angoff standard-setting methods. Applied Measurement in Education 1999; 12 (2): 151- 65. [DOI:10.1207/s15324818ame1202_3]
26. Pellegrino JW, Jones LR, Mitchell KJ. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. National Academy Press; 1999.
27. Shepard LA. Implications for standard setting of the National Academy of Education evaluation of the National Assessment of Educational Progress achievement levels. InProceedings of the Joint Conference on Standard Setting for Large-Scale Assessments 1995; 2: 143-160.
28. Abbasi H. [Comprehensive assessment and determination of the standards of mastery in specialized examinations for entry into internship courses in the medical field using the classic and featured model] [dissertation] .Allame TabaTabaei University; 2013. [Prsian]
29. Linacre JM. Winsteps Rasch measurement computer program user's guide. Beaverton, Oregon: Winsteps. Com; 2012.
30. Minaei A. [Application of Rash Measurement Model in Evaluation of Measurement Measurement Features of Visual-Motion Skills Test (TVMS-R)]. Educational Measurement Quarterly 2015; 5 (18): 77-114.[ Persian]
31. Yudkowsky R, Downing SM, Wirth S. Simpler standards for local performance examinations: the Yes/No Angoff and whole-test Ebel. Teach Learn Med 2008; 20(3): 212-7. [DOI:10.1080/10401330802199450]
32. Peterson CH, Schulz EM, Engelhard Jr G. Reliability and validity of bookmark‐based methods for standard setting: comparisons to Angoff‐based methods in the National Assessment of Educational Progress. Educational Measurement: Issues and Practice 2011;30(2):3-14. [DOI:10.1111/j.1745-3992.2011.00200.x]
33. Mortaz Hejri S, Jalili M, Labaf A. [Setting Standard Threshold Scores for an Objective Structured Clinical Examination using Angoff Method and Assessing the Impact of Reality Chacking and Discussion on Actual Scores].Iranian Journal of Medical Education. 2012;11(8):885-94.[ Persian]
Send email to the article author



XML   Persian Abstract   Print


Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:

Habibzadeh S, Delavar A, Farrokhi N, Minaei A, Jalili M. The Use of Rasch and Item Mapping in Determining Cut Score of Comprehensive Pre internship Exam. RME 2019; 11 (3) :59-70
URL: http://rme.gums.ac.ir/article-1-794-en.html


Rights and permissions
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Volume 11, Issue 3 (2019) Back to browse issues page
پژوهش در آموزش علوم پزشکی Research in Medical Education