Assessing Students Problem Solving Skills (Part 2)

The following assessment methods are used to assess clinical problem solving:
1. The script concordance test (SCT) assesses clinical reasoning in the context of uncertainty. Because there is no single correct answer, scoring is based on a comparison of answers provided by examinees with those provided by members of a panel of reference made up of experienced practitioners.
2. Observations of complex problem solving can be arranged using audio/video recordings of patient interactions or by acting as a neutral observer. Video recording is particularly useful because student’s can self evaluate and will have less opportunity for the “he said, she said” defense. Checklist and narrative assessments would be useful for recording the observations.
3. Problem differentiation cases use examples of history and examination findings that are similar except for one or two key points. This is useful to assess the student’s pattern recognition skills.
4. Self-reporting inventories such as the Problem Solving Inventory (PSI) can be useful in assessing the student’s ability to recognize the level of their skills.
5. Use the RIME method to assess where the student is performing on a 4-point scale (Reporter, Investigator. Manager, Evaluator).
Other Resources
Evaluation of Teaching and Learning Strategies

Assessing Students Problem Solving Skills (Part 1)

The following case illustrates how good problem solving requires more than medical knowledge.
An 18-year-old student and her boyfriend come to emergency because she is having trouble breathing. She has a history of viral asthma and her parents had always accompanied her in previous emergencies. Upon examination, the resident determines that she is not having an asthma attack and takes the boyfriend aside and tells him she is faking and walks away. A month later, she arrives by ambulance, unconscious after trying to walk to the hospital alone because her boyfriend assumed again that she was faking and refused to take her to the hospital.
Let’s assume that the initial diagnosis of “not an asthma attack” is correct, where did the resident fail in his diagnostic and therapeutic management?
Attribution error: The resident’s use of the term “faking” says a lot about why he overlooked talking to the patient about whether stress (leaving home, exams, boyfriends) had led to contracting an irritated airway and did not suggest she find a family doctor for management of her asthma. If she was indeed faking, then he didn’t make any attempt to identify why because attribution is a common reason for making the value judgement that someone is not worthy of thorough care.
Ethical error: The resident told the young man the patient was faking, but did not tell her what he thought.
When designing an assessment plan, the following steps should be followed:
1. Review the components of a useful assessment plan
2. Review the objectives for the rotation
3. Determine what specifically your plan will test
4. Identify assessment tools
5. Plan how students and college will receive results
6. Train the administrators
7. Evaluate utility of the plan
1. Review the components of a useful assessment plan
According to Dr. Gordon Page from the UBC College of Medicine, his research has shown that the Utility or usefulness of an assessment plan can be expressed by the following formula: U = R X V X E X A X C
Reliability is the result of increasing preceptor experience with a tool and evidence that the device tests what it is supposed to test. A single instance of student performance is not considered reliable in medical training.
Validity in a clinical setting is the result of testing for application of knowledge not recall, diagnostic and therapeutic reasoning not thoroughness, and an adequate sampling of behaviour (6-10 observations).
Educational impact is a combination of the impact this testing device has on the student, preceptor and institution.
Acceptability by the student, preceptors and institute is a key factor of ongoing utilization.
Cost is the final element.
A combination of paper cases and direct observation meets the above utility requirements better than cases or observation alone.
2. Review the objectives for the rotation
Problem solving objectives should have been included in the student orientation to the rotation. As previously discussed, an example can be found here
3. Determine what specifically your plan will test
Ask yourself and colleagues “What behaviour will tell me that a student has achieved the objectives?” The more specific and observable the behaviour the easier it will be to test. Ex. Student uses appropriate social and cultural criteria when making a therapeutic diagnosis rather than Student doesn’t make attribution errors.
4. Identify assessment tools
If you need to create assessment tools rather than using ready-made instruments, please consult with the assessment specialist in Educational Support and Development or some other group with experience in creating assessment devices.
5. Plan how students and college will receive results
If assessment is being done primarily to improve learning, then a feedback process needs to be determined. Written and verbal feedback are both useful at this stage. If you are assessing competence at the end of a rotation, then a more formal process needs to be determined.
6. Train the administrators
Observation of problem solving and giving feedback are skills that faculty may feel inadequately prepared for to administer.
7. Evaluate utility of the plan
The best laid plans ……
Stay tuned for the next installment on testing methods.

Digital Doctoring

Thanks to Dr. Premkumar for the following information.
Dr. Joseph Martin, the dean of Harvard Medical College, authored an essay in the Boston Globe this week, entitled “Digital doctoring”. Dr. Martin writes, “A generation ago, doctors were taught that they were all-knowing healers whose judgment was sacrosanct. But today, there’s simply too much to know. With the overwhelming advancement of innovative drugs and procedures, doctoring has moved from an individual endeavor to a team effort, and it is technology that binds the team together.” He concludes, “Eventually, of course, physician resistance to technology will fade as the generations turn. Until then, it is important that the revolution continue, for technology benefits everyone — by lowering costs and saving lives.”

Medical Problem Solving Part II: Teaching Problem Solving

First year Family Medicine residents were able to arrive at the correct diagnosis for only 25% of the presented (complex) problems – Norman, 1994

Clinicians approach problems flexibly; the method they select depends upon the perceived characteristics of the problem. Complex problems need a combination of three strategies:

  1. Pattern recognition (illness scripts, schema) is the result of years of clinical experience; this recognition is associated with higher rates of successful diagnosis (Coderre, 2003.)
  2. Forward-based reasoning works well for routine and well-researched medical problems. Standard clinical teaching follows a pattern whereby several diagnostic hypotheses are generated from the patient’s signs and symptoms, then additional information is sought to confirm or refute individual hypotheses.
  3. Backward reasoning means guessing at an explanation and working backward to find data that supports the explanation. Novice doctors have a tendency to use backward reasoning and should be encouraged to use this strategy for non-routine, complex cases but not those that are routine and/or well-researched.

These three thinking strategies are the most commonly used clinical problem solving strategies because they create a constant regenerating of new hypotheses to test.
Steps for Teaching Problem Solving
Step 1: Identify the problem solving objectives for your student
The information on the following website might be useful for identifying learning objectives
Step 2: Reinforce problem-solving steps in clinical settings

Researching the problem
Consistently use a Subjective (what is the patient saying, what is the family saying), followed by Objective (what does the examination and tests tell you) process for patient presentation, so the student becomes used to thoroughly investigating the problem, both scientifically and affectively/culturally. Having the student draw a decision tree may also be useful in helping them identify gaps in their investigation.
Teach students to use close-ended questions (What medication are you taking?) for diagnosing standard problems and open-ended questions (What issues is the abdominal pain causing in your day-to-day life?) for more complex problems.
Question the student to help them pinpoint gaps in their approach: “Are their other systems involved? Is there something you missed or are unclear about in the initial examination? What other information do you need that isn’t readily available? What was difficult about this patient? What assumptions are you making (possible attribution errors) about this patient?”
Differential diagnosis
You can assist students to begin to recognize patterns as well as recall previously learned information by involving them in a large number of cases, thinking out loud so the student sees your pattern making and asking questions like “What were the major findings that led you to this conclusion? What else could it be? Have you seen anything similar before? Is there something different about this case from other cases you have seen or read? What is the pattern here? How would you categorize this problem? If xxx changed, how would you approach the problem?”
Selecting criteria for possible treatment options
Ask the student “Given what you know about the patient subjectively and objectively what should be considered before suggesting a treatment plan?” There are factors that influence compliance with treatment that physicians need to consider before deciding on a therapeutic diagnosis that novice doctors may not be aware of such as the following:

  • affordability
  • resources in the home community or neighborhood
  • cultural or religious practices
  • age
  • support systems
  • other medical issues

Help the student to check the validity of any biases that might result in attribution errors (an unjustified tendency to assume that a person’s actions depend on what "kind" of person that individual is rather than on the social and environmental forces that influence the person).
Therapeutic diagnosis
Use One Minute Preceptor technique to clarify the student’s decision-making process.
Evaluating the solution
Metacognition or the process of thinking about how one problem solves is an important step in improving student skills. Metacognition is the result of two components in the learning environment: written reflection and probing questions from peers and instructors. Questions such as the following can be useful in creating a more reflective mind set: “What did you learn from this patient? How could you have involved other people (family, community groups, nurses) in the care of this patient?”

Step 3: Place students in situations were the routine approach will fail (without endangering the patient or the student)
Students need opportunities to be challenged by cases that are more complex or difficult to define. They also need opportunities to correct mistakes safely because it is through practicing non-routine decision-making that critical thinking skills are honed. On the other hand, student and patient physical and emotional safety must be ensured if you want the student to continue to increase their skills.
Coderre, S. et all, Diagnostic reasoning and diagnostic success Volume 37, Number 8, August 2003, pp. 695-703.
Norman, G.R. et all, Cognitive Differences in Clinical Reasoning, Teaching and Learning in Medicine, Number 6, 1994 , pp.114–20.
Web Resources
Teaching students to think like physicians
Teaching problem solving and decision-making
Internal Medicine Clinical Teaching Unit: Is it an Inappropriate Site for Student Learning?
View box exercises for teaching problem solving

Death of a child from AIDS

“Tonn” is a video about the death of a orphan in a hospice for terminal HIV patients in Thailand. This movie realized by the doctor of that hospice, is a way to pay homage to the child and to promote the wearing of condoms.
Images and topic are very graphic. View with caution.

Companion video that tells a spouse story is available at the story of two young people going for testing is here

Medical Problem Solving – Part 1: Stumbling points for students

This is the first in a series of postings about teaching medical problem solving.
The following five steps in problem solving can each create unique stumbling points for students and novice doctors:

  1. Researching the problem
  2. Differential diagnosis
  3. Selecting criteria for possible treatment options
  4. Therapeutic diagnosis
  5. Evaluating the solution

Researching the problem

Medical history taking, examination and reviewing test results are the standard first steps in patient problem solving. Two common mistakes can be made at this point:

  1. not asking the right questions – Novice physicians are particularly prone to not asking the right questions because they do not have the vast storehouse of experience to make connections in new or unique situations. They may also not have a clear understanding of affective or cultural influences in the patient’s life.
  2. listening in order to record rather than with a problem-solving ear.

Differential diagnosis

The differential diagnosis process based on medical evidence for standard medical problems works well. Every supervising clinician should be aware of the following issues that can lead to medical errors:

  1. attribution errors – an unjustified tendency to assume that a person’s actions depend on what "kind" of person that person is rather than on the social and environmental forces that influence the person. An example from the book How Doctors Think – a woman comes in severely underweight; the physician thinks woman + underweight = anorexia and proceeds under that assumption without identifying the celiac disease that was the problem. Another example would be the drug abuser who tells an untrue story about what brought them to the emergency room; the physician thinks drug addict + lie = manipulating for more drugs and the physician misses seeing a brain injury.
  2. assuming that scientific evidence is applicable beyond the parameters of the study. For example, studies done on men are assumed to be applicable to women or studies done on one race are assumed applicable to all races.

Selecting criteria for possible treatment options for this individual

There are factors that influence compliance with treatment that physicians need to consider that novice doctors may not be aware of such as the following:

  1. affordability
  2. resources in the home community or neighborhood
  3. cultural or religious practices
  4. age
  5. support systems
  6. other medical issues

(It is important to avoid attribution errors by not assuming a person is a stereotype.)

Therapeutic diagnosis

Novice doctors may run into difficulty in the following situations:

  1. a previous diagnosis is assumed to be correct, therefore previous treatment is continued
  2. the patient has complex problems combined with multiple therapies by multiple practitioners
  3. the patient follows homeopathic or other health practices which may not be disclose under questioning such as “what drugs are you taking?”

Evaluating the solution

Ongoing reflection is an important skill for physicians, but students may not have an opportunity to receive information on what happened to patients after their brief encounter. This is particularly problematic when sub-optimal care results in short term solutions that create long-term issues that the student is not aware of.

Another issue that arises occasionally for students is the lack of resources available to deal with medical error in a constructive solution oriented way rather than punishment oriented (self or authority).

Using Active Learning

Active learning is a teaching method that resulted from studies about how the brain stores information. Students that interacted with content through a combination of reading, writing, drawing, discussing and reflecting were more likely to remember that content than did students who passively listened to what the faculty member said. Most active learning techniques take less than five minutes of class time. To listen to a broadcast about active learning, see my website
Active learning techniques that could be used in medical education include:
1. The “One Minute Paper”
2. Class wikipedia (online encyclopedia or definitions)
3. Write, pair, share
4. Muddiest (or Clearest) point
5. Daily journal
6. Compare notes
7. Reading quiz
8. Clarification pauses
9. Scavenger hunts
10. Student summary of another student’s answer
11. Deposit questions in The Fish Bowl
12. Flash cards
13. Puzzles/Paradoxes
14. Evaluation of another student’s work
15. Active review sessions
16. Concept mapping
17. Visual lists
18. Role playing
19. Panel discussions
20. Debates
21. Educational games
For more information about these and other active learning techniques, sign up for the Educational Support and Development series on Active Learning.
Some websites that explain these techniques in more depth include:
Some websites with resources that might be useful for creating active learning include:
Concept Mapping
Crossword Puzzle Maker
Medical Mnemonics
Online Games Generator
Thiagi Game site
Usask Blogs
Usask Wiki

Profile #1 Dr. Tom Wilson

On March 29, 2007, I had the great pleasure to interview Dr. Tom Wilson who has been the Joint Head of the Department of Medicine for the University of Saskatchewan and the Saskatoon Health Region since July 1, 2003. Dr Wilson graduated from medical school in 1967. He completed the Teaching Improvement Project System (TIPS) course in 1986.
1. Looking back to your training as a physician, was there a particular teacher who influenced your approach to medical education?
Dr. Alan Gilbert was an early hero of mine. He practiced Internal Medicine in Moose Jaw, and then moved to Alberta to join the Faculty of Medicine there. When he interacted with patients, he was professional yet friendly; sometimes he joked with them or used touch to reassure. He taught me the importance of really listening to patients and doing a complete physical exam before deciding what was wrong. I remember him looking at a picture of Jackie Gleason’s (The Honeymooners) palms and commenting that he probably had liver problems. Several years later Jackie Gleason died from liver cirrhosis.
I knew I wanted to be internist after having him as my teacher. He taught us through professional respect, never talked down to us and was quick to praise. He didn’t disparage us as students, but helped us see mistakes. He would fit in very well in today’s medical school.
2. What changes have you seen in medical education since you trained as a physician in the late 60’s?
a. Medical knowledge has expanded and there are medical disciplines that didn’t exist 30 years ago. The internet has made access to medical literature faster and easier to access. More and more we see medicine as a lifelong learning process, so we need to teach students how to actively involve themselves in learning.
b. It is critical to keep up to date on changes in medical education. For example, technology such as PowerPoint has changed how we teach. Faculty today have an opportunity to learn how to teach, something that wasn’t true in the past. I try to keep up to date by taking teaching courses and reading.
c. Residency has changed from slave labour to more of a learner – teacher environment.
3. What change would you like to see in medical education in the next ten years?
a. Earlier streaming for people who know what they want to do. We need a shortened, more efficient streaming process. Twelve years can be very discouraging.
b. More training in interdisciplinary teams because of the shortage of physician’s and the rising cost of health care.
c. A representative workforce
4. What area of medical education would you like to learn more about?
a. active learning
b. problem solving
c. brief bedside encounters as a method of evaluating JURSI’s and residents
I would like to thank Dr. Wilson for being the first of a series of interviews.

How doctors think – Intro

I recently had the pleasure of a first appointment with a specialist about some health issues I’ve been dealing with. As part of his history taking, he asked me questions about general health and surgeries (pretty standard stuff). The difference was he really listened to what I was saying from a problem solving (not a recording perspective) and came up with a diagnosis for a chronic health problem that had never been suggested before even though I had told 7 doctors in the last 10 years about my concerns. He then gave me material to read about his diagnosis and I went and did some research on my own that supported what he had told me.
Contrast this experience with one I had in my early twenties where I went to my doctor with severe stomach cramps. The physician introduced me to the young doctor he was teaching as another woman with mysterious stomach pain. Beyond my frustration at being typecast in that manner, I knew what was wrong but was not being listened to. A week later, I had emergency surgery for the very issue I had tried to explain. This doctor failed me as a patient but he also failed as a role model for his student.
I was of course personally interested in the specialist’s ability to diagnose, but I was also professionally interested in how we teach differential diagnosis. I recently ordered a book called “How doctors think” Dr. Groopman believes that some preventable medical errors are a direct result of not listening to the patient because we place them into categories such as women with mysterious abdominal pain (attribution errors) or rely on diagnoses from previous medical professionals that may have been incomplete.
I plan on doing a series of posts from this book when it arrives.