Assessing Students Problem Solving Skills (Part 1)

Case
The following case illustrates how good problem solving requires more than medical knowledge.
An 18-year-old student and her boyfriend come to emergency because she is having trouble breathing. She has a history of viral asthma and her parents had always accompanied her in previous emergencies. Upon examination, the resident determines that she is not having an asthma attack and takes the boyfriend aside and tells him she is faking and walks away. A month later, she arrives by ambulance, unconscious after trying to walk to the hospital alone because her boyfriend assumed again that she was faking and refused to take her to the hospital.
Let’s assume that the initial diagnosis of “not an asthma attack” is correct, where did the resident fail in his diagnostic and therapeutic management?
Attribution error: The resident’s use of the term “faking” says a lot about why he overlooked talking to the patient about whether stress (leaving home, exams, boyfriends) had led to contracting an irritated airway and did not suggest she find a family doctor for management of her asthma. If she was indeed faking, then he didn’t make any attempt to identify why because attribution is a common reason for making the value judgement that someone is not worthy of thorough care.
Ethical error: The resident told the young man the patient was faking, but did not tell her what he thought.
When designing an assessment plan, the following steps should be followed:
1. Review the components of a useful assessment plan
2. Review the objectives for the rotation
3. Determine what specifically your plan will test
4. Identify assessment tools
5. Plan how students and college will receive results
6. Train the administrators
7. Evaluate utility of the plan
1. Review the components of a useful assessment plan
According to Dr. Gordon Page from the UBC College of Medicine, his research has shown that the Utility or usefulness of an assessment plan can be expressed by the following formula: U = R X V X E X A X C
Reliability is the result of increasing preceptor experience with a tool and evidence that the device tests what it is supposed to test. A single instance of student performance is not considered reliable in medical training.
Validity in a clinical setting is the result of testing for application of knowledge not recall, diagnostic and therapeutic reasoning not thoroughness, and an adequate sampling of behaviour (6-10 observations).
Educational impact is a combination of the impact this testing device has on the student, preceptor and institution.
Acceptability by the student, preceptors and institute is a key factor of ongoing utilization.
Cost is the final element.
A combination of paper cases and direct observation meets the above utility requirements better than cases or observation alone.
2. Review the objectives for the rotation
Problem solving objectives should have been included in the student orientation to the rotation. As previously discussed, an example can be found here http://www.hsc.stonybrook.edu/som/solving.cfm.
3. Determine what specifically your plan will test
Ask yourself and colleagues “What behaviour will tell me that a student has achieved the objectives?” The more specific and observable the behaviour the easier it will be to test. Ex. Student uses appropriate social and cultural criteria when making a therapeutic diagnosis rather than Student doesn’t make attribution errors.
4. Identify assessment tools
If you need to create assessment tools rather than using ready-made instruments, please consult with the assessment specialist in Educational Support and Development or some other group with experience in creating assessment devices.
5. Plan how students and college will receive results
If assessment is being done primarily to improve learning, then a feedback process needs to be determined. Written and verbal feedback are both useful at this stage. If you are assessing competence at the end of a rotation, then a more formal process needs to be determined.
6. Train the administrators
Observation of problem solving and giving feedback are skills that faculty may feel inadequately prepared for to administer.
7. Evaluate utility of the plan
The best laid plans ……
Stay tuned for the next installment on testing methods.

1 thought on “Assessing Students Problem Solving Skills (Part 1)

Leave a Reply

Your email address will not be published. Required fields are marked *