Category Archives: Assessment and Evaluation

Sequencing an outcome

Posted on by

How do you breakdown a course outcome into manageable sections, or chunks, of information or action for students? What do students need to be able to do with automaticity before they can meet this learning outcome? Here is a template to help you in docx form:

Sequencing Template and Mini-Lesson Template

Sequencing Template

A. My end goal is to have students be able to meet this specific learning outcome:
B. Therefore, I need students to be able to do these three (or more) sub-components, in this sequence below C. I will get them to practice and build their abilities in these subcomponents by doing these isolated activities. %
1) ->  
2) ->  
3) ->  
D. Now that they have practiced these distinct and separate skills, I will have them comprehensively do this activity:  
E. I will assess the cumulative work with this assessment tool:    

More information on sequencing and ‘chunking’ here:

Course Development Toolkit

https://teaching.usask.ca/curriculum/online-delivery.php#CourseDevelopmentToolkit

Course Design: Implement and Evaluate

https://teaching.usask.ca/articles/course-design.php

How to get students to hand in quality work by planning for choice

Posted on by

In my course, at this level, at this place of progress in their learning, what do students need to demonstrate to me?

Handout version for USask Instructors

What do I expect of my students?

Offering choice in how students meet course objectives is rooted in inclusive education and that by providing choice we acknowledge and respect that there are many ways to demonstrate learning and students have the agency, when appropriate, to pick the one that motivates them. These checklists might help you think about “shifting the ‘locus of control’ from the teacher to the student” (Jopp & Cohen, 2020)

There are three methods described: when students pick the medium of transmission for a final project, when students pick the topic of a paper or structured project, and lastly, when students help set the criteria for the assessment. The last one is my favourite because it forces the students to really think about what ‘good’ means, not just the content to transmit.

Allow students to choose the format of their final project – follow these steps

  • I have reviewed the outcomes in my course.
  • I have determined the outcomes that are best suited for offering choice.
  • I can think of a few ways a student could show me how they meet this specific outcome.
  • I have considered how students might present a portfolio of work to present their learning of the outcome.
  • I can make ONE assessment tool (such as a rubric or checklist), such that no matter which option students choose, I can use the same marking scheme. AACU Values Rubrics are a good starting place https://www.aacu.org/value-rubrics
  • All options would demonstrate understanding to a similar depth and breadth of disciplinary ability.

Have students choose the topic for their project or paper

  • I have a date when the assignment needs to be completed.
  • I have a date when a first draft should be completed.
  • I have a plan for when and how students will give each other feedback on their progress.
  • I have determined when I want to review students’ project plans – before they get too far down the road to make significant changes.
  • I have an idea of how much time I need to allocate to review students’ work or what my review supports look like (TAs, assistants, mini-interviews).
  • I have a plan for how I will introduce the project, its criteria, the options for students, the marking scheme (rubric or checklist), and present the timeline with deliverables to students.
  • I have a timeline of all of the above and am ready to share it with students.

Co-create the rubric or assessment tool with students

  • I know what ‘good enough’ looks like.
  • I know what ‘not good enough’ looks like.
  • I have anonymized examples of sufficient and insufficient that I could show students.
  • Students know the small steps or pieces that compose this larger assignment and can describe them.
  • I ask students for their feedback about what they think is important in an assignment.
  • I ask students what they think would qualify as good.
  • I work with students to make a checklist of what would be required to qualify as ‘good’ (criteria).
  • I am reflecting on how to use a checklist, rubric, or other marking guideline to communicate my expectations with students.

References:

Jopp, R., & Cohen, J. (2020). Choose your own assessment – assessment choice for students in online higher education. Teaching in Higher Education, 1-18.

https://www.utsc.utoronto.ca/technology/assignment-scaffolding

https://clas.ucdenver.edu/writing-center/sites/default/files/attached-files/scaffolding_assignments.pdf

 

Leveraging Peer Feedback in an Online Environment

Posted on by

When students take courses in-person, they often find at least one friend in the course with which they discuss the course, the assignments, give feedback to one another, and so on. With the shift to remote delivery, students might have more trouble finding someone to connect with on their own.

This post explores how you can introduce peer feedback into your course to ensure that your students have a chance to share their work and receive feedback from peers. In the best cases, they might even form friendships, but another benefit of using peer feedback is that the quality of student work usually increases, which can make your marking much more enjoyable.

There are a few different approaches that could be taken to implement peer feedback, but the most common method is to use the discussion board. Below are the steps of what you’ll want to think through if you’re planning to use the discussion board for this:

  1. The first thing is to decide on what assignment(s) you want to include peer feedback.
  2. The second step is to determine a reasonable timeline for the peer feedback. This can be tricky in spring and summer courses, as assignment due dates usually come fast and furious! For example, if you want the assignment submitted to you on Friday, students should be sharing their drafts for peer feedback no later than Wednesday (but Monday or Tuesday would be best). There needs to be time for peers to provide the feedback and for the student to implement the feedback before submitting.
  3. The third step is to decide on which students will provide feedback to one another. It is overwhelming to ask students to review the work of all of their classmates. Instead, you will want to assign pairs or small groups. You can implement this in your course by either formally using the Groups tool in your LMS (e.g., Blackboard) or you can create forums for each pair or group. There are pros and cons to both approaches
    1. If you use the formal Groups tool, only students in each group will see the assignments, discussion, and feedback. This limits the ability to see a really wide variety of their peers’ work
    2. If you create different forums for your “groups” on the course discussion board, all students will still be able to see each other’s assignments, but should be instructed to only provide feedback to their group. Since everyone can see the feedback, students may be more guarded in their feedback
  4. The fourth step is to decide the format and instructions for the peer feedback
    1. Do you have a rubric created for the assignment? If yes, it would be great to ask the students to use the rubric when providing feedback
    2. Do you have a list of assessment criteria or expectations? In this case, formulate questions or areas of focus for the peers to consider when providing peer feedback
    3. Do you want the students to provide their own areas of focus for what they specifically want feedback on? You could instruct students to post their assignment along with questions for their peers to consider
  5. The last step is to implement this planning within your LMS! Please contact the GMCTL or Distance Education Unit (DEU) for support.

Worried About Take Home Exams and Academic Misconduct?

Posted on by

There is more than one way of looking at the extended time period for open book exams when it comes to academic misconduct worries.   

You can see 24 hours of unsupervised time with an exam as more time for students to break your rules.   Some students may use the time that way and that is deeply frustrating.  

More encouraging is a view based in research about academic integrity in higher education (for a great review, see the 2013 book referenced below). The basic premise is this:  When students are more confident they can do what needs to be done on their own, they are less likely to cheat.  

Obvious?  

Perhaps less obvious is how this 24-hour, open book exam format may allow for this:  

  • Students are not being tested on how quickly they can answer questions.  Speedy students are always advantaged in short time frame exams and there can be many reasons some students aren’t fast at writing exams that are unrelated to what they know or can do.  If you are more interested in correct than fast, this format allows students to show you what they know and they may feel more confident as a result.
  • Students do not need to strategize about allocating their time to the same extent.  Effective traditional exam writers do this to maximize their scores given the time limits.  Under time constraints, areas of uncertainty could have been where the worried student had the cheat note, or looked at someone else’s exam, or called a friend. Now, students will have more time to work on the areas of uncertainty, and they may feel more confident as a result.
  • Students can concentrate better with breaks – mental and physical.  The human attention span is said to be 20 minutes, plus, humans should only sit for 30 minutes at a time.  Effective traditional exam writers probably pause and reset mentally at least, but many students think they just have to keep going as fast as possible.  Students will rightly think they can think better when breaks are more possible – and they may feel more confident as a result.

And, one bonus point to ponder about open book exams, unrelated to time frame:

  • Students do not become preoccupied with memorizing answers because it is not necessary to do so.  Memorized material doesn’t last in long term memory anyway (unless they use it 6 more times). Students recognize the exam will be more about how to use the information, or solve the problems, or apply the theory, or analyze or diagnose the case, or evaluate potential alternatives, and so on. These are higher order cognitive abilities and allow students to show greater depth of understanding.  If students work on these abilities to prepare for an exam, what they learn is much more likely to last. This is how the brain works.  When students know the exam does not require memorization, they may feel more confident as a result.

 These above points may not apply perfectly to all disciplines, teaching styles, or assessment philosophies.  However, if you watch to see if these advantages of this format arise, maybe you’ll opt for an exam something like this from now on? 

Lang, J. (2013).  Cheating Lessons:  Learning from Academic Dishonesty.  Harvard University Press: Cambridge, MA.

It Helps To Be Transparent About Academic Integrity

Posted on by

You and your students will be out there wondering how fair final assessments can be when everyone is unsupervised.  Thankfully, there are some students, that no matter what, will follow the rules and maintain their academic integrity.   A small number, will seek to cheat no matter what we do.   The group to focus on right now is that large majority that wants the rules to be clear, to be enforced, and for there to be a level playing field for all.  The majority of students want to be honest, but at the same time, they do not want to feel at a disadvantage if they are.

 When students see their instructor making specific efforts to protect against academic misconduct, many become more committed to academic integrity as a result.   Here are example statements that might be helpful:   

 Designing the assessment:

  • “I have designed the final assessment to ask you to apply what you have learned, using more personalized examples, rather than questions that ask you to recall the facts or recognize correct answers.  This makes copying answers less relevant.” 
  •  “I have designed the assessment expecting you will access your course materials.  I have created variation in the questions that each student receives, so that exchanging answers does not work for students who would cheat in this way.”

 Reducing pressure students feel:   

  • “I have reduced the weighting of the final assessment to lower the stakes for you during this anxious time.  I know students under pressure sometimes take shortcuts or make blatant academic misconduct decisions and I do not want that to happen to you.”
  • “The university policy for take home finals has reduced the time pressure compared to traditional exams.  Also, if something goes wrong with your technology or you are feeling unwell,  you can take breaks and come back to the exam.  I know students under pressure sometimes take shortcuts or make blatant academic misconduct decisions and I do not want that to happen to you.”

 Creating more student agency and responsibility:  

  • “I have provided you with information about the questions in advance, so that you know more about how to prepare for the exam.  When there are fewer “mysteries” about the exam, usually students show more academic integrity, and that’s what I want.” 
  • “I have given you some choice in format/topic area for the final assessment.  When students get to choose, I expect you will show me your best work and feel more confident about doing it honestly, and that’s what I want.”  

 Expressing your and students’ commitment:   

  • “I have asked all students to sign and return this document (e.g., see declaration of secrecy, or Sample Open Book Exam Academic Integrity statement) as a further step in acknowledging the importance of individual commitment to academic integrity.  If I find students have engaged in academic misconduct, this form can also be used to question students further about their understanding of the rules”.
  • “I will follow up directly with students who have submitted work that raises suspicions of academic misconduct.  I will follow up, in part, out of respect for all the students who did their work with academic integrity under these unexpected conditions.  If your work concerns me,  you can expect me to require an online or phone meeting to discuss. I will follow my College’s procedures and University of Saskatchewan policy.”

Tips From Veterans of Remote / Online Teaching

Posted on by

As you prepare to create and teach courses remotely this spring and summer terms, we asked some U of S instructors experienced with this type of teaching to share some quick words of wisdom based on what they learned from their own experiences. Below are their tips related to design, teaching, and assessment for remote / online. Thank you to Jorden Cummings (Psychology), Allison Fairbairn (Music), Hayley Hesseln (Agriculture and Bio Resources), and Karla Panchuk (Geology) for sharing your experiences teaching remote / online courses.

Design

  • Keep it as simple as possible. Online learning difficult for many students for diverse reasons, and we cannot assume our students have access to all things technology or that they are fully comfortable using technology.
  • Make it as easy as possible for students to find what they need within the course. Karla Panchuk shared this screen shot with us to demonstrate an easy to use sidebar menu in Blackboard.


Teaching

  • Pre-record what you need to, but there are numerous existing resources you may find useful (e.g. YouTube videos)
  • Provide students with weekly checklists that highlight key items they need to look at
  • Post slides with voice-over. Post the notes page as well.
  • Use discussion groups to facilitate students connecting
  • Ask students to introduce themselves and say something that nobody knows about them
  • Ask students questions during the slide presentation/recording
    • Post those questions in a discussion forum to promote dialogue
    • Ask students to post a comment and reply to comments to encourage participation
    • Use the discussion forum with guided questions for readings
  • If you use outside services, avoid ones that require your students to create an account
  • Limit the number of tools that you and / or students will need to learn to use. Never assume that your students will all be tech-savvy

Assessment

  • Have clear rubrics and post them so students can see how you will assess their work ahead of time
  • If feasible, provide a peer review component that  allows students to receive additional feedback on their work
  • If students need to print materials off to fill out and then submit, they need to be allowed to take photos of these to submit since a scanner might not be available
  • Setting up the grading centre to do anything, but the basics is time consuming. Just download your grades and calculate offline
  • Hayley Hesseln also makes use of reflections on learning as assessments

How much help are students getting on their assignments?

Posted on by

You may be concerned with how much feedback or advice students are getting on their assignments or open exams.

Below are some options, reasons to try them, and some tools for implementing.

Option Reason Helpful tools for this
Require an acknowledgement of feedback, guidance or teachings received
  • Respectful of contributions of others
  • Common academic practice as seen in many published papers
  • Truthful

 
  • An example acknowledgement from a paper you have written,
  • An example where you have been acknowledged;
  • Another example that you find or create that could fit with the assessment
Distinguish types of feedback and their acceptability: e.g., proofreading, editing, error correction, peer teaching, conceptual changes…
  • Communicates differences that students may not see in what makes some help ok and other help not ok
  • Define e.g., proofreading vs. editing vs. re-writing for your students
  • Restate the rules for the assessment with respect to thi
  • Reiterate the follow up you will take re suspected academic misconduct
Make getting feedback a requirement  (from a classmate, or someone else)
  • Shows you know how valuable feedback is for learning
  • Points students to the criteria for the assessment
  • May mean you get better work overall (probably fewer little errors)
  • You can assign peer feedback partners
  • Provide a rubric or a checklist for the assignment that students use to give each other feedback
  • Sample acknowledgement as in the first option in this table

 

 

How to Support Students Who Have Multiple Final Exams In 24 Hours

Posted on by

Giving students 24 hours to complete a final assessment might make a lot of sense in a time of emergency, but it can also cause anxiety for students if they are unclear on expectations. You can help by clarifying expectations and stating clear limits in advance. To assist all of your students, but particularly students facing this additional challenge, please include the following in your exam information to students:

  • Put all instructions about the exam into the introduction of the exam to ensure that your students have easy and clear access to any instructions, including logistics and what they should do if they have trouble submitting the exam. If you have already provided these instructions elsewhere, repeat them within the introduction to the exam.
  • Provide students with an estimate of how much time the exam will likely take (i.e. I expect it will take you about three hours to write this exam)
  • Provide students with clear requirements related to their answers to short answer and essay questions, such as minimum and/or maximum word counts for answers. This will also help you manage the time needed for marking.

Be aware of and provide students with needed supports and accommodations required from Access and Equity Services (AES). You can find information about meeting these requirements during the COVID-19  pandemic on the AES website.

If you know of specific students in your classes who are scheduled to have two exams within a 24 hour period you could:

  • Email deu.exams@usask.ca to verify if a conflict exists for a student requesting a change. For more information on this see the FAQ on our Final Exams and Course Assessments page.
  • Discuss with those individual students what specific additional needs they may have, if any.
  • Negotiate an earlier 24 hour period of time for the release of the open book exam
  • Offer those students a bit of extra time, such as a 48 hour window

Ways To Limit Concerns About Non-permitted Collaboration

Posted on by

Are you worried about non-permitted collaboration?

It’s true, students can seek each other out for help, examples, interpretations, translations, feedback, and peer teaching when unsupervised. In fact, we often encourage students to do so as part of the learning process.  Blatant “copying” is a real problem because then the submitted work does not represent what that individual student knows or can do “without the support of resources or colleagues.”  Most students want assessment to be fair and are likely to appreciate your effort to have everyone play by the same rules.

Below are two options you may want to consider for addressing concerns about students collaborating.

Option 1:   Limit collusion by design.

  • Make collusion more difficult.  Develop more than one version of the exam recognizing you will need to deal with some administrative implications.  The variation of exams will make it more cumbersome for students to work together.
  • Make collusion less applicable.  Design the questions on the exam to require unique answers relevant to the student.  Ask questions that cannot be answered the same way by several students.
  • Articulate a process to verify students’ answers. For an example approach see, Learn how to use an oral exam (5-10 minutes per student).
  • Request students submit a signed document attesting to their honesty.  A recent post includes a sample (at bottom of post) that may cause students to think more carefully about their actions.
  • For more information on design, Download an explanation of how to transform your current exam questions into open-book questions

Option 2:  Permit collaboration and build it into your assessments.

  • Have students describe their process for consulting resources or people.
  • Provide a rubric that shows the criteria for levels of use of resources and feedback.
  • See this example of a worksheet (bottom of the post) or student pairs who provided each other with feedback as part of an individual paper assignment, this could be adapted for finalizing an open book exam.

 

Assessment Equity and Alignment with Experiential Learning

Posted on by

When I met with Sandy and Harold I was stressed. I was worried that I was falling behind. After coming from a very busy workplace with many competing deadlines and defined work hours, starting a PhD program and having to manage my time independently is a huge challenge. Most days feel chaotic and I’m often overwhelmed. Being a student has given me space, mentally and emotionally, to think and to focus on my health. But this “room to think” can also be a dangerous thing. Sometimes hours, even days, slip by in an unfocused haze of meandering reading if I’m not careful. This skill of balancing time and energy is the true test of graduate school.

The chaos is exacerbated by the scope of my SOTL research. I often feel that I’m lost in a forest and blind to the connections, discourses, and secret paths within the literature. What has helped is staying in the meadow of multiple-choice questions. As I read more, I am beginning to create a complex picture of this learning and assessment modality. But before I made this decision, I wandered, as you’ll see.

I was getting tired of assessment, so I experimented with other search terms to see what I could find. I found Brunig’s article which brings together experiential education and critical pedagogy, both of which are challenging to implement in practice. Their discussion of their own practice is very helpful for those interested in using experiential learning and developing a student-directed classroom. Brunig makes a claim in their article that one of the aims of experiential education is the development of a more just world (p. 107). I was intrigued by this because I didn’t necessarily agree (although I believe this should be the goal of education in general). They referenced another paper when making this assertion which brought me to Itin. Itin’s article attempts to differentiate between experiential learning and experiential education. This paper is a good foundation for instructors who want to explore the philosophy of experiential education.

After this foray into more theoretical papers I decided to hold off on reading any more until I had spoken to Sandy and Harold. I wasn’t sure how far the group wanted me to go when it came to philosophy and theory. This was a good decision because the group agreed that they want to see more practical research and studies than theory.

The other articles can be summarized:

  • Bowen, C. discusses a college mathematics program at Haskell Indian Nations University. Of interest to mathematics educators are the handful of experiential activities provided by Bowen. Most of the activities are not suitable for large or mega class sizes (although there is the possibility of adaptation). Others such as narrative word problems or the Problem of the Week may be useful with larger classes.
  • Butler, A. C., Karpicke, J. D., & Roediger, H. L. 3rd clearly describe the methodology used. Results of this study indicate that “delayed feedback produced better long-term retention than immediate feedback” (p. 279). What is interesting is that “there was no difference between the two types of feedback” (answer-until-correct and standard – correct or incorrect) (p. 279).
  • Clandinin, D. J. & Connelly, F. M. are two of the originators of narrative inquiry and their book offers researchers new to the methodology a guide to follow. From epistemological concerns to the nuts and bolts of how to actually do a narrative inquiry, this handbook is a wonderful starting point for those interested in this methodology.
  • Ernst, B., & Steinhauser, M. suggest that the P300 and early frontal positivity “are related to two different stages of learning. The P300 reflects a fast learning process based on working memory processes. In contrast, the frontal positivity reflects an attentional orienting response that precedes slower learning of correct response information” (p. 334).
  • Koretsky, M. D., Brooks, B. J., & Higgins, A. Z. very clearly outline their research design and methodology. Their findings “suggest that asking students for written explanations helps their thinking and learning, and we encourage instructors to solicit written explanations when they use multiple-choice concept questions”, p. 1761.

 

References

Bowen, C. (2010). Indians can do math. In P. Boyer (Ed.), Ancient wisdom, modern science: The integration of Native knowledge in math and science at tribally controlled colleges and universities (pp. 43-62). Salish Kootenai College Press.

Breunig, M. (2005). Turning experiential education and critical pedagogy theory into praxis. Journal of Experiential Education, 28(2), pp. 106-122.

Butler, A. C., Karpicke, J. D., & Roediger, H. L. 3rd. (2007). The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied 13(4), pp. 273-281.

Clandinin, D. J. & Connelly, F. M. (2000). Narrative inquiry: Experience and story in qualitative research. Jossey-Bass.

Ernst, B., & Steinhauser, M. (2012). Feedback-related brain activity predicts learning from feedback in multiple-choice testing. Cogn Affect Behav Neurosci, 12(2), 323-336. doi:10.3758/s13415-012-0087-9

Itin, C. M. (1999). Reasserting the philosophy of experiential education as a vehicle for change in the 21st century. The Journal of Experiential Education 22(2), pp. 91-98.

Koretsky, M. D., Brooks, B. J., & Higgins, A. Z. (2016). Written justifications to multiple-choice concept questions during active learning in class, 38(11), pp. 1747-1765. doi: 10.1080/09500693.2016.1214303

This is part of a series of blog posts by Lindsay Tarnowetzki. Their research assistantship is funded by and reports to the Scholarship of Teaching & Learning Aligning assessment and experiential learning cluster at USask.

Lindsay Tarnowetzki is a PhD student in the College of Education. They completed their Master’s degree at Concordia University in Communication (Media) Studies and Undergraduate degree in English at the University of Saskatchewan. They worked at the Clinical Learning Resource Centre at the University of Saskatchewan for three years as a Simulated Patient Educator. They are interested in narrative and as it relates to social power structures. Lindsay shares a home with their brother and their cats Peachy Keen and MacKenzie.

Image provided by Lindsay.