University of Nottingham - Moving from OMR to CBA for summative exams in medicine

Moving from OMR to CBA for summative exams in medicine

Background & Context
University of Nottingham has very large medical programmes (one 5 year programme and one 4 year graduate programme with 1,900 students). Students need to do a range of ‘high stakes’ assessments/exams some true/false/abstain question type questionnaire exams need also to be scanned into a computer to download into hard copy this scanning takes time in addition to the marking. There was also a wish to make the tests more realistic and interactive utilising interactive question types such as drag and drop labelling and image hotspots.

Implementation & Technology
There was no steer from the university as a whole and the department could have used a number of software as the university supports Blackboard, Questionmark Perception and other WebCT applications. However the department chose to use Touchstone in 2003 and it was decided to stay with this as there was experience within the department on its application both from the student and staff perspective.


Issues
· Students being able to google or use of a pen-drive for the answers
· Students actually seeing other student’s computers due to the layout of the rooms
· Large scale exam sitting that are depended on IT could fail and need to be secure for such high stake examinations


Issues overcome
· Large computer lab accommodating 150 students who sit back to back and one smaller lab both used simultaneously
· An 'exam desktop' was created. This used group policies in Windows XP to restrict access to no applications apart from Internet Explorer and no web site apart from TouchStone used for examination purposes. It also restricts access to any drives and other operating systems


Stakeholders involved in the process
· Academics creating the questions,
· Administrators performing room booking/timetabling,
· Subject matter experts conducting standards setting reviews,
· External examiners,
· Disability experts and
· IT support personnel


Tangible Benefits
· Student’s records could be kept in one place and reviewed electronically and quickly.
· Time to create and implement and mark the exam is reduced from hours to seconds, student receive results faster and staff have more time to review students who are appearing to fall behind.
· Exam questions and be reviewed and reformed fast and effectively
· The interactive nature of the computer environment allows for a more realistic impression for students, student therefore gain a better understanding of their topic.
· Students with disabilities have adjusted interfaces that enabling better understanding of questions and automatic time added where appropriate.
· Banks of questions can be formed and cascaded between years and medical disciplines, and easier changed
· Log on details could be forwarded to the external examiner who could review the process, questions and sample the answers at random so reducing the administrative and paperwork for review.


Lessons learnt
· All stakeholders need to be involved from the outside to ensure large cohorts are able to take the exams fairly, quickly with results processed efficiently and effectively
· Implementation should be undertaken in stages (cohorts, subject areas etc) to minimise problems and maximise learning opportunities
· Use of soft wear is important but so it is the ability of the staff to support this. Thus Touchstone was the preferred programme.

Kathryn’s thoughts
This type of course and the way it is assessed is completely new to me and my closes experience is the diving exams I take that are multiple choice questions (and I hate them). I can see that such high stakes exams need to be carefully planned and executed and I can also see that from a student perspective they need the results back quickly.

I think the teams acknowledgment of the steps involved is good and also the fact they note the time saved is used to identify students who many need additional support. They have also thought about the aspect of accessibility at the onset and this too is essential.

What I think about the whole process is that I see that this is training at a fast and frantic level that perhaps teaches medical students to conform from an early stage – they are just part of a huge system (the NHS) and they are only small porns within that system.