- How useful was this post?
- Item Writing Manual
- Best Practices on Examination Construction, Administration, and Feedback
- National board of medical examiners item writing manual book
- Successful Test Item Writing
- Somatic Dysfunction in Osteopathic Family Medicine
- We are sorry that this post was not useful for you!
- Item Writing 101: Bloom’s Taxonomy
- Constructing Written Test Questions For the Basic and Clinical Sciences
- Teaching/Learning/Evaluation Tools for Health Professions Educators
Examinations are typically used in higher education to objectively assess student learning, and they are also used as a frequent assessment tool in the Doctor of Pharmacy curriculum. This paper describes best practices and provides examples for faculty to build reliable and valid examinations, ensure examination security and deter academic misconduct, and enhance student learning and achievement of course objectives. Colleges and schools of pharmacy can incorporate these concepts into comprehensive examination policies and focus faculty development efforts on improving the examination purpose, design, and experience for both faculty and students.
Examinations are a frequent assessment method used in higher education to objectively measure student competency in attaining course learning objectives. The Accreditation Council for Pharmacy Education standards hold pharmacy programs accountable to ensure the validity of individual student assessments and integrity of student work. Additionally, uniform agreement is lacking on the best procedures for examination administration, and whether examinations should be returned to students or retained by faculty.
This commentary provides an overview for best practices in examination construction and blueprinting, considerations for ensuring optimal and secure administration of examinations, and guidance for examination reviews and feedback on student performance. For an examination to be psychometrically sound, it must be reliable and valid.
How useful was this post?
An examination is considered reliable if the results it generates are consistent and reproducible, such that a student would perform similarly on multiple versions of the examination. The goal of validity for examinations is to ensure that a representative sample of the intended learning objectives is measured, and that students have satisfied the minimum performance level to be competent with respect to the stated objectives.
There are four important principles that faculty need to consider when creating a content-valid examination. First, establish the purpose of the examination and take steps to ensure that it measures the desired construct s. Second, link items on the examination to the course learning objectives and the intended teaching taxonomies.
Third, ensure that items are clearly written and well-structured; items that are ambiguous or lack congruence with the objectives may confuse students and directly affect examination scores. The last principle specifies that experts in the field should review the examination to ensure the other three principles have been met.
Backward course design is an effective method that helps faculty determine which learning objectives, outcomes, and competencies should be assessed and how. When using examinations as an assessment strategy, faculty must consider which learning objectives are best assessed using this method.
Item Writing Manual
Examination blueprinting is one method for faculty to ensure that their assessments align with the intended student learning objectives and levels of learning, and should be completed prior to examination construction. When blueprinting, faculty must also consider logistics such as the total number of items and overall length of the examination.
Time spent on each item will vary based on the level of difficulty of the question, as well as the volume of reading associated with the question.
For example, it takes much less time for a student to answer a question with a simple stem based on strict memorization of fact than to read a long and detailed patient case, think critically about the information, and synthesize an answer. Faculty should consider only including auxiliary information when truly needed to answer the question; it can be frustrating for students to spend examination time reading a long case only to be asked a question they could have answered without it.
Faculty should use blueprinting to reduce two key validity threats: construct representation and construct-irrelevant variance. Issues with construct representation more specifically misrepresentation, underrepresentation, or overrepresentation occur when there is under-, over-, or biased sampling of the selected examination content.
Faculty can minimize this threat to validity by ensuring proportional representation of content covered within the examination. The blueprint for item allocation and overall weighting should be proportional to the coverage of course content. In a simple scenario where all topics are considered equally important and are delivered using similar instructional methodologies, examination items should be divided among topics proportionally.
Best Practices on Examination Construction, Administration, and Feedback
Faculty can support the validity of their examination by planning properly for construct representation. Construct-irrelevant variance CIV occurs when there are flawed item formats, such as when examination items are written at an inappropriate level of difficulty eg, too hard or too easy or the wrong question format is chosen. CIV is often seen as a systematic error that either inflates or deflates a test score, which introduces bias.
Using a range of difficulty minimally difficult to very difficult is helpful to ensure that the examination can be completed within the allotted time. This is particularly important for examinations with questions from multiple instructors. The last step in the examination construction is test item development. It is helpful for new faculty to field-test items before they are used on examinations, and to use peer expert reviewers to establish validity of items.
After the assessment is created, it requires administration under proper conditions. Policies for examination administration should be designed with input from both faculty and students to outline expectations for student conduct and provide protocols for security to minimize opportunities for dishonesty.
Well-developed policies should also identify penalties associated with policy infractions.
National board of medical examiners item writing manual book
Unfortunately, when it comes to cheating, cost ie, potential punishment versus benefit ie, better grades is sometimes an ethical dilemma for students. Ip and colleagues found that It is wise for faculty and programs to assume that dishonesty will occur without concerted prevention efforts; consistency among faculty and administrators regarding setting expectations, and reinforcing the examination policy and code of conduct, creates a culture of academic integrity.
The first step in ensuring examination security is to maximize attendance. At institutions with multiple campuses, examinations should be administered during the same time period. Examination times should ideally be scheduled to avoid conflict with commonly attended professional events, to avoid the need for make-up examinations.
Policies limiting the scope of acceptable absences and requiring advanced notification with documentation for anticipated absences eg, meeting brochure may serve as a deterrent. In the case where absence cannot be avoided, acquiring identification is key unless the student is well known to the proctor to prevent dishonesty through use of a surrogate.
To maximize examination integrity, makeup examinations should be administered after the regularly scheduled examination for the course is administered. Testing the same outcomes on a make-up examination through different assessment questions or in a different fashion eg, essay instead of multiple choice may reduce requests for make-ups to those absolutely necessary.
However, having a different make-up test has several implications such as perception of fairness to all students eg, extra study time or less desirable examination formats , faculty workload, and scheduling challenges when large numbers of students are excused from the initial test date. A creative measure might include administration of the make-up examination with a proctor off-site, such as when a faculty member attends the same conference with students.
Remote proctoring services using technology are also available. Students must be on time for examinations, both for security purposes and to prevent distracting others who have started the examination.
Many programs either set an established cut-off time for entry or employ policies that prohibit entry of new examination takers after the first examination taker has left the room. Though in theory this practice seems logical if one assumes that examination security has been compromised, unless a different examination is offered for makeup, it may make more sense to have the student begin the examination rather than delay further. Penalties such as having less time to complete the examination or automatic point deductions may motivate a tardy student for future timeliness.
For electronic examinations, late arrivals require students to open their examinations to ensure they did not begin their examination after unauthorized receipt of a password from someone already in the testing room.
Successful Test Item Writing
Students should not be excused from the examination room without good reason; only one student at a time should be permitted to exit and only after ensuring their materials are secure.
The number of trained proctors should be chosen to maximize security and limit distractions. It is also good practice to include at least one faculty member to assist in supervision and assume responsibility in the case of cheating or unforeseen emergencies.
Somatic Dysfunction in Osteopathic Family Medicine
The faculty proctor need not have material on the examination to proctor, since answering questions during the examination may create an unfair advantage between students. Proctors should receive specific training in how to establish and maintain a secure examination environment, as well as handle dishonesty and other emergent situations. The most important aspect of such training is that they avoid distraction and take the task at hand seriously. Too often, proctors including faculty proctors, may be found sitting with their heads down, looking at their phones, or grading papers.
Proctors, or invigilators ie, one who is vigilant , should walk up and down the aisles and make eye contact with students.
Remote examination proctoring is becoming available through a variety of vendors for use during large examination administrations, with distance-learning students, for make-up examinations, or other instances when it is difficult to secure skilled proctors.
In remote proctoring, students take examinations while monitored using web-cam computer technology, as well as fingerprint scanners to verify student identity.
We are sorry that this post was not useful for you!
Remote proctoring has been shown to serve as a deterrent of cheating, but requires further investigation. Random examination seating either by chart or at the direction of proctors may enhance security. Seating charts also allow for seats to be blocked eg, back rows to ensure easy movement and viewing by proctors.
To decrease the burden of randomizing every time, faculty should prepare several versions of a seating chart for each cohort in advance and choose the final version at exam time.
Seating charts may also aid in confirming attendance. For additional security and verification of attendance, students may be asked for identification, to wear ID badges, sign an attendance log, or even provide biometric confirmation.
Item Writing 101: Bloom’s Taxonomy
Another strategy to deter cheating is through administration of different versions of the examination. With paper examinations, avoid using different color paper or large font headings indicating examination version to prevent collusion between students with the same version. Electronic testing allows for randomization of sequence and item choices. Programs using electronic testing may consider the use of privacy screens as an additional, but not foolproof, security measure.
If used, it is recommended that specifications and estimated cost for screens are included in the technology requirement policy; there is great variation in type, cost, and viewing angle limits of available options on the market.
Careful consideration should be paid to restricting items from students at their seats to those essential for taking the examination eg, sharpened No. Bulky coats, hoodies, and hats may be restricted with exceptions made for religious purposes , to prevent students from hiding notes or other items. Even the simplest of items may be suspect including pens, mechanical pencils, tissue packets, food, and drinks, which is also why board examinations do not permit these items.
Faculty should consider restriction of watches and activity monitors entirely, communicating time via room clock or electronic testing software. This alleviates the need for proctors to assess whether each item is suspect, and prevents notes from being hidden beneath the face or band.
Similarly, distribution of non-programmable calculators or use of electronic examination calculators is recommended.
Constructing Written Test Questions For the Basic and Clinical Sciences
Lastly, faculty should strongly consider creation of and distribution of uniform reference materials eg, calculation formulas and scratch paper ideally of an alternate color , so proctors may easily identify them. All restrictions should be outlined within the exam administration policy to ensure consistency in expectations by students and proctors.
Periodic conversations, surveys, or focus groups with students may further inform faculty and administrators of how effective these measures are at deterring cheating. All students are expected to adhere to the Code of Conduct of their program, which should address matters of cheating including aiding someone else as well as the inappropriate use of technology resources.
Students should take measures to protect their own work and be cautious of behaviors that give the appearance of cheating eg, talking, wandering eyes, possessing restricted items.
Having a secure examination environment and good proctoring may deter cheating; however, both students and proctors who witness or become aware of acts of academic dishonesty during an examination or examination review session should be encouraged to report the concern. Students should alert a proctor of suspicious behavior discretely but immediately, so the situation may be assessed, ideally with the assistance of a faculty member.
Of note, Rabi and colleagues determined that classroom atmosphere can influence cheating behaviors and suggested that faculty who appear more approachable and less intimidating can reduce cheating in the classroom.
Perhaps the best way to deter cheating is to reinforce expectations and apply punishment for infractions. Faculty should take seriously any reports of cheating on examinations, or in any assessment. Sharing de-identified statistics regarding reports of cheating and resultant consequences lends credibility to the importance of reporting for all stakeholders. Those who are honest and do not cheat are left feeling discouraged and their efforts devalued.
Teaching/Learning/Evaluation Tools for Health Professions Educators
Interestingly, for cases of cheating, nursing students most frequently suggested receipt of a zero on the assessment, and expulsion from either the program or university. Some institutions have policies and procedures regarding student reviews of completed examinations, and whether students may keep the examinations.
In the absence of institutional policy, faculty often debate the pros and cons of returning examinations to students. However, unless there is no reasonable way for the student to access their records, this right does not entitle them to receive a permanent copy. Both the pros and cons of returning examinations must be considered.