Category Archives: Assessment and Evaluation

Top Hat: How is it being used at the U of S?

Posted on by
Reply

The University of Saskatchewan has a continuing commitment to a technology-enhanced learning environment for students and in January 2016 acquired a campus-wide license for the Top Hat student response system. Top Hat is a software-based student response system, incorporating a “bring-your-own-device” solution, that is available at no direct cost to instructors and students. The primary goal of Top Hat is to enhance the teaching and learning experience for both instructors and students by bringing more engagement and interaction into traditional passive lecture-style learning approaches.

Who we are

We are a research team at the University of Saskatchewan who are interested in student response systems with a specific focus on Top Hat, their pedagogical effectiveness, and investigating the best teaching practices for these systems. Our team is organized under the Scholarship of Teaching and Learning (SoTL) cluster titled “Technology-Enhanced Learning: An Assessment of Student Response Systems in the University Classroom.”

  • Carleigh Brady, PhD, Instructor, Dept. of English
  • Soo Kim, PhD, B.Sc.PT, Professor, School of Rehabilitation Science
  • Landon Baillie, PhD, Professor, Dept. of Anatomy, Physiology and Pharmacology
  • Raymond Spiteri, PhD, Professor, Dept. of Computer Science
  • Neil Chilton, PhD, Professor, Dept. of Biology
  • Katherina Lebedeva, PhD, Instructor, Dept. of Anatomy, Physiology and Pharmacology

In March of 2018, we invited all individuals with a Top Hat instructor account at the University of Saskatchewan to participate in a survey about the use of Top Hat on campus and to share their experiences.

Results

 A total of 58 instructors responded to the survey. We found the majority of instructors using Top Hat at the University of Saskatchewan:

  • incorporate it in class to assess student concept understanding, test student recall, and share student perspectives (opinions, experience, and demographics)
  • use it for asking questions, creating discussions, and monitoring attendance
  • prefer “multiple choice question,” “word answer,” and “click on target” formats
  • think that the greatest advantages of Top Hat are: increased participation and engagement, student assessment, instant feedback from students, and the system’s ease of use/functionality
  • consider that Top Hat’s biggest disadvantages to be the time investment for software setting-up and grading, design issues, and technical issues (e.g. room connectivity)

In summary, we found that most instructors using Top Hat found it effective in facilitating a collaborative teaching and learning environment. Top Hat encourages students to participate actively during lectures by asking questions and polling student responses online. Despite some disadvantages, Top Hat is still preferred over clickers for its increased functionality (various question formats, interactive functions, and use of graphics), as well as its instant feedback and results polling.

However, further studies should be conducted to systematically evaluate the effect of Top Hat on student academic performance.

 Find more information

Is Your Instruction Designed to Produce Student Learning?

Posted on by
Reply

Lecture is an efficient way to transmit information, especially in large classes. We inevitably feel there is a lot of content to cover, since the gap between what novice students know and expert professors know is large. However, large, uninterrupted blocks of lecture are very inefficient ways to learn, because they are passive. Learners get cognitive overload and stop processing, have trouble paying attention, and remember some ideas that they struggle to apply or connect conceptually.  All of these occur, even with strong learners, and even with instructors who provide exceptionally focused, clear delivery of information. The mind just learns more if it is actively engaged in thinking.

As a method of direct instruction, lecture is focused on a well-organized, clear presentation of information.  Its cousin, explicit instruction, is much better aligned with what we know about how the brain learns, because it is active.

Explicit instruction:

  • students are guided through the learning process with clear statements about the purpose of learning the new skill
  • teachers give clear explanations and demonstrations
  • students engage in supported practice with feedback at intervals throughout the entire class, not just at the end

The key distinction is that while there are periods of telling information, student are asked to demonstrate the skill they are learning and practice it with feedback.  As a result, they are much more likely to remember, make fewer errors, are more focused, and more motivated.  They are also more likely to describe the learning and important and describe how to keep improving. There is clear alignment between the goal of having students understand more deeply, and the activities they are asked to participate in to support their learning.

Why does all this mater?

When we set the goals for what our students will be able to know and do by the end of class (outcomes), we need to think carefully about how remembering information is essential, but not sufficient, learning. We want students to be able to correctly apply the new information in a process, to make decisions and informed judgments, and to use new information for reasoned arguments.  That means our classes need help students develop these competencies and practice them with feedback. Our outcomes, our instruction, and how we assess all need align and work cohesively together to support effective learning processes. If they don’t, we become Professor Dancelot of video fame, with good intentions and little actual learning.

Learn more:

  • Interesting Book: Donald A. Bligh, What’s the Use of Lectures? (San Francisco: Jossey-Bass, 2000), pp. 252, 11.
  • Oft cited scholarly history: C. Bane, “The Lecture vs. the Class-Discussion Method of College Teaching,” School and Society 21 (1925); B.S. Bloom, “Thought-Processes in Lectures and Discussions,” Journal of General Education 7 (1953).

It’s All About Your Outcomes

Posted on by
Reply

Structurally, outcomes are obligations. You need outcomes for your course syllabus, and your program as whole has some form of outcomes. From a teaching and learning perspective, however, an outcome is much more than just a hoop.  It’s at heart of why you’d bother to teach the course you do. Each outcome (and you don’t need that many), describes a skill, disposition, or set of complex knowledge that it is essential for your students to demonstrate to be successful in the course.

What does a good outcome look like?

You can read more about definitions of outcomes (what a student will do) and objectives (what an educator will teach) in another post from Gwenna Moss, but sometimes good examples can help clarify a definition.  A good outcome satisfies key criteria, including:

  • It starts with a specific, rigorous verb that reflects the type of thinking, attitude, or understanding you need students to demonstrate
  • Each outcome is worthy enough that you’ll spend a good chunk of the course returning to it and building your students’ strength with it
  • The outcome is written in language students understand and can explain in their own words

A bad outcome: Understand the definition of a just society

This outcomes is not good because there are too many ways the word “understand” could be interpreted. What would be good evidence of an outcome should be easy for students to understand the same way.  Also, this outcome might be able to be satisfied with a definition from the professor’s PowerPoint, so it isn’t worthy and long lasting enough. It is easy to make the mistake of basically describing content in your outcome, rather than what your students will demonstrate.

A much better outcome: Justify arguments about social justice using precise, accurate examples.

This is better because it specifies the type of thinking and skills student will need to do (justify an argument) and at how (using precise, accurate examples).  Social justice is a complex concept that the course will spend a long time on, deepening student conceptual knowledge overtime. Also, the skill of building an argument about social justice will built upon many multiple times in the course, sometimes in class discussions, sometime in an essay, and sometimes in an examination.  A student reads the outcome and knows the course will help you refine your skills in building arguments, and that the content will relate to social justice.

How do I write good outcomes?

  1. List the key concepts, skills, and dispositions/attitudes you’ll want in the course.  Check to ensure you aren’t just listing content.
  2. Group related things together until you have a smaller number of bigger things.
  3. Try writing statements describing things you’d accept as evidence that a student actually had the understanding, skill, or disposition you are trying to teach
  4. Look at the statements you’ve written, and ensure they each start with an active, specific verb.  Try using this list to ensure you are asking for rigorous thinking, not something students can just memorize and forget.
  5. Get someone who is not an expert to read each outcome and tell you what it says, just to make sure it is clear enough
  6. Double check that each outcome represents something you’ll want to see from students multiple times in the class. If you wouldn’t want to grade things related to it more than once, it is not important enough to be an outcome.

 

Promoting Academic Integrity: Some design questions for instructors

Posted on by
Reply

[social_share/]


Here are some propositions about students’ academic integrity that I’ve been working with:

  1. Students are more likely to do their work honestly when they see the personal value in what is to be learned.
  2. Students are more likely to do their work honestly when they believe the assessment produces actual evidence of what they have learned.
  3. Students are more likely to do their work honestly when they’ve had the chance for practice and feedback.
  4. Students are more likely to do their work honestly when they know the rules and expect them to be enforced.

Designing assessments for academic integrity is much more than tight invigilation processes and tools like Turnitin and SafeAssign (thankfully). There is much that instructors can do to set students on honest learning paths when they design and teach their courses.   Below, I offer some prompting questions instructors can ask themselves when designing course materials, assessments, and learning activities that relate to the four propositions above.

“See-the-Value” Questions for Instructors:

  • How can I convey/demonstrate the value of what students learn in my course?
  • How can I share my enthusiasm for learning this and the value I place on it?
  • How can I connect students to the benefits this learning brings for them individually, for their families or communities, for society or the world?
  • How can I provide opportunities for students to follow their individual interests and values in the context of this course?

“Evidence” Questions for Instructors:

  • What kind of evidence does this assessment provide that students have learned what I wanted them to learn?
  • What other forms of evidence could I use to determine this?
  • What alternatives could I offer students to show me what they have learned?
  • How can I make explicit to students that an assessment is transferrable to other contexts?

“Practice and Feedback” Questions for Instructors:

  • What do students need to be able self-assess their progress before grades are at stake?
  • How can I provide early feedback so that students still have the opportunity to improve?
  • How can I stage larger assignments with feedback so that students have time to improve (and avoid last minute temptations)?
  • How could I best equip students to provide feedback to each other?

“Rules” Questions for Instructors:

  • What are my rules for my assessments within the academic integrity policy framework?
  • How can I clearly explain both the assessments and the rules so that students know how to best proceed?
  • What are some common misconceptions/errors that I can address early on?
  • How can I help students learn how to follow the rules, especially when it involves technical components like a new citation or referencing protocol?
  • How can I show students that I am committed to enforce my own rules?

We have a workshop coming up at the GMCTL on November 14 that will explore assessment practices that promote academic integrity. Please consider registering.

When Performing Gets in the Way of Improving

Posted on by
Reply

[social_share/]


I encountered the following video in the spring and have been sharing it with faculty and groups with an interest in questions of assessment.  I think it lays a useful foundation for discussions on (1) what it takes to master skills and knowledge, (2) the value of lower stakes practice, (3) the necessity of formative feedback for learning, and (4) recognition that moments of “performance” or assessment for grades are also needed.

Additionally, this video supports the thinking behind a core element of the Instructional Skills Workshop—an internationally recognized workshop and certification offered regularly at our Centre.  For that workshop, participants practice the facilitation of a 10 minute “mini lesson” so valuable for improving instructional skills.  Here’s a link to more information about that workshop.

Happy to discuss the learning zone and the performance zone with Educatus readers!

First-time Thoughts on a Student Blog Assignment

Posted on by
Reply

[social_share/]


By Yin Liu, Associate Professor, Department of English
History and Future of the Book Blog

Why I did it

In 2016-2017 I taught, for the first time, a full-year (6 credit unit) English course, “History and Future of the Book,” which is one of our Foundations courses – that is, it is one of a few 200-level courses required for our majors. As in all our courses, there is a substantial writing component, usually in the form of essay assignments. I decided to complicate my life further by trying out a type of student assignment also new to me: a student-written course blog.

I had been thinking about using a student blog assignment ever since I heard a talk given by Daniel Paul O’Donnell (U Lethbridge) about using blogs in his own teaching. The point that struck me most forcibly about Dan’s argument was his observation that students wrote better when they were blogging. Since one of my goals in teaching writing is to help students write better, I thought I should give the idea a try.

Setting it up

From the outset, I had to make some fundamental decisions about how the blog assignment should work within the course. It became one of the writing assignments, taking the place of a regular (2000-word) essay: the blog post itself was to be 500-1000 words long and accompanied by a commentary (read by me only) in which students discussed the process of writing the blog post, especially the challenges they encountered and the solutions they developed. The commentary gave students a chance to reflect on and thus to learn from their own writing processes; it also helped me to evaluate the effectiveness of the assignment. The blog was made publicly available on the Web, but students could opt out of having their own work posted, although it still needed to be submitted to me for grading. Thus students also needed to supply signed permission to have their work published on the blog.

Heather Ross of the GMCTL guided me to the U of S blog service (words.usask.ca) and gave me valuable advice about permission forms and other such matters. The ICT people set me up and increased my storage quota, I fiddled with the WordPress templates, and we were good to go.

Results and learning

Each student wrote one post for the course blog, and thus the assignment was like a regular term paper except that (a) it was not an essay, and (b) it was published to the Web. Acting as the blog editor, I suggested changes to students’ first submissions, which they could incorporate into the final, published version if they wished; but I resisted the temptation to tinker with their final versions, which were published warts and all. I also used the course blog to post a series of Writing Tips for the class.

Students did, for the most part, write noticeably better in their blog posts than in their regular essay assignments. More was at stake in the blog posts; students knew that their work would be read not solely by their professor, but also by their peers and possibly by others outside the class. The informal nature of a blog also allowed students to write, in many cases, with a more genuine voice than for an essay assignment, and thus more effectively. This less formulaic, less familiar genre compelled students to rethink the basics of writing: ideas, information, audience, organisation, clarity. There was a higher chance that they would write about something that truly interested them, and quite a few expressed enthusiasm about the assignment. Students could also read and learn from the work of others in the class. The experiment was a success, and I would do it again.

Our course blog, History and Future of the Book, can be found at https://words.usask.ca/historyofthebook/. Some of the students’ posts have been removed at their request, but most remain, and you are welcome to browse through the Archive and read them – the best of them are excellent.

Open Pedagogy: Using OER to change how we teach

Posted on by
Reply

[social_share/]



There has been a considerable increase in the number of courses assigning open rather than commercial textbooks at the University of Saskatchewan.  During the 2014-2015 academic year, there were approximately 300 students enrolled in three courses using open textbooks. This year more than 2,650 students are enrolled in the at least 20 courses that have open textbooks as the assigned resource. Since the university started promoting and tracking the use of open textbooks in 2014, this use has resulted in students at the U of S saving close to $400,000 on textbook costs.

The benefits of using open textbooks and other open educational resources (OER) instead of commercial texts aren’t limited to the cost savings for students, however. The lack of copyright restrictions on OER allows instructors to modify these materials to meet the specific needs of their courses. For example, the Edwards School of Business recently released an adaption of an open book from the United States that not only saved their students money, but also meets the learning needs of the students better than the original edition. University Success will be used by the more than 475 students in the course, but also students at other institutions, and other instructors will be free to make their own changes to this resource to better meet local needs.

Just as instructors are able to adapt existing OER, so are students. Learners can become contributors to existing open materials, or use OER to create new learning materials for themselves, their peers, and future learners (and instructors).

John Kleefeld, a professor in the U of S College of Law, created an assignment that offers students in his The Art of Judgement course the chance to improve Wikipedia articles on one of the topics covered in the course. Professor Kleefeld and one of those students, Katelyn Rattray wrote an article on the design of the assignment and the experience that was published the Journal of Legal Education.

Robin DeRosa from Plymouth State University in New Hampshire created an open textbook for her early American literature course by having undergraduate research assistants find appropriate public domain content. As a core assignment in the class, students then wrote introductions for each reading based on their research about the authors and time periods.  While she served as the editor, students did much of the research and compiling of content for this new open textbook. This assignment replaced a traditional paper that would have only been seen by the instructor and the student and likely soon after marking, discarded by the student. Read more about this process on her blog.

Moving away from private “throw away” assignments can shift student activity away from knowledge consumption instead developing their skills in knowledge creation.  In the examples above and many others, this lead to increased student engagement, improved learning outcomes, and freed instructors from reading the same assignments repeatedly.

If you would like to learn more about open pedagogy, the GMCTE is offering a session on November 8 as part of our Introduction to Learning Technologies series.  You can also contact the GMCTE directly with any questions or to schedule a consultation.

Ideas about Assessing Student Participation

Posted on by
1

[social_share/]



Recently we completed another Instructional Skills Workshop (ISW) at our Centre.  This an intensive and engaging 4-day workshop where faculty and instructors learn about and practice participatory learning strategies, and upon completion, receive a certificate of completion that is nationally recognized.

As the workshop unfolds, important questions are brought forward by participants.   Given our focus on student participation in the ISW, the question of how to (and whether to) give participation marks arises.  While the answers depend on the context of the course, the teaching approach, and the design of the learning experiences and assessments, specific ideas from others can help us arrive at ways of doing this that can fit our individual teaching.

So, when I came upon a link to this blog post by David Gooblar –  I thought it was worth highlighting to our recent ISW graduates, and to readers of Educatus.

In particular Gooblar links to an article by Tony Docan-Morgan that includes a participation log template that can be found here.

We would welcome more ideas from our readers on this teaching topic.

First Day of Class: Providing students a relevant and engaging initial taste

Posted on by
Reply

[social_share/]



Sessions related to this topic will be held during the Fall Fortnight:

  • Why Teach With Top Hat? (Monday, August 22, 2016 from 10-10:25 AM) – Register Here
  • Building Student Capacity for Effective Group Work (Monday, August 22, 2016 from 1-4 PM) – Register Here 
  • Preparing & Personalizing Your Syllabus (Tuesday, August 23, 2016 from 1-2:50 PM) – Register here
  • Exploring Methods for Preventing & Detecting Plagiarism (Wednesday, August 24, 2016 from 10-11:30 AM) – Register Here
  • Attention & Memory: Increasing Student’ Learning (Friday, August 26, 2016 from 9-10 AM) – Register Here
  • Assignments, Rubrics, and Grading in Blackboard – It’s Easier Than You Think (Thursday, September 1, 2016 from 3-4:30 PM) – Register Here

As people, our perceptions and routines are engrained early on for places and people. Our experiences and decisions shaped by earlier ones. Find yourself or your students returning to their same seat? Initially excellent (or poor) meals at a restaurant tinting later dining experiences? Buying a book when the first few sentences catch your attention?

The first day of class is similar: it sets the focus of the course; instills interest; sets the context including layout and participation levels; invites selecting (or dropping) a class; and builds initial credibility and approachability.

As the recent Chronicle Vitae post by Kevin Gannon (Director of the Center for Excellence in Teaching & Learning and Professor of History at Grand View University) noted just reading the syllabus and ending class is not enough. You miss modeling how you expect students to engage in the classroom, and missing the opportunity for intriguing questions and enthusiasm for the topic. As Kevin Gannon notes “Whatever your plan for the first day, students should get some idea of what’s expected of them throughout the semester, and also have the opportunity to discern their place in the class and its activities.”

One example of engaging students in the content of the course and the level of participation and thinking you expect to see, is highlighting one of the essential ideas of the course in a way that is immediately relevant to students. Create that initial, individual, motivating connection to what they will learn.

keyholeAllow your students to glimpse the potential and be curious about where it will lead.

Highlighting a core focus of the course

1) Identify key principles, lessons or concepts that are foundational to your course

  • Resources: decoding the disciplines & threshold concept literature in nearly every discipline offers ideas to select from
  • Stats class example: ability to read descriptions of quantitative research to identify and critique reported and missing components of data description and analysis
  • Qualitative research class example: all forms of reporting involve speaking for the other person, including choosing what is conveyed and how.

2) Convey the hook of curiosity & why it is inherently meaningful to them.

  • Tip: Each discipline arose to explore and work on essential and important inquiry. As experts in the discipline, the importance is obvious but often hard to articulate to a novice.
  • Resources: Reflect on what makes this topic inherently so important and relevant. Not sure how to describe it? Try talking it out with a colleague outside your discipline (or one of us in the GMCTE).
  • Stats class example: a sample online news report of a relevant hot topic based on “research”
  • Qualitative research class example: meet and then introduce a classmate

3) Connect what students already know with what they will learn

  • Tip: Having a clear framework early on in a course allows students to organize their new ways of thinking and new information. Referring to the example, diagram, key ideas throughout the course reinforce them and help to encode new memories.
  • Resource: The meaning step in 4MAT approach to lesson planning. Attention and memory literature. There is an upcoming fortnight session on Friday August 26th.
  • Stats class example: identifying the statistical & research pieces in the news story provided and missing pieces. The initial step for critiquing.
  • Qualitative research class example: experiencing the sense of responsibility and uncertainty when speaking for another person (especially when they are sitting beside you). Wondering did they say enough or too much? Were they accurate or misinterpreted? And then connecting it with the key idea of self & research within qualitative research methods.

Photograph courtesy of David Hetre under a CC-BY license.

ePortfolios and the Curious Case of the End-of-Term Journal

Posted on by
Reply

[social_share/]


Sessions on this topic will be held during the Fall Fortnight:

  • Mahara ePortfolios (Short & Snappy session) (Monday August 22, 2016 from 11- 11:25 AM) – Register here
  • Mahara ePortfolios (Expansion Pack session)  (Tuesday August 23 from 10:30 – 11:50 PM) – Register here

As an undergrad, I took a senior studio art class in which I had to contribute something, anything, daily (well, at least weekly) to a visual journal we would hand in at the end of term. I did nothing with that journal until a stressful and long two days before it was due. My prof loved the hastily complied and craftily “aged” journal I submitted. I even pressed aged-looking coffee cup rings onto some of the pages. However, I would have gained far more from the course had I taken the time to truly focus, reflect, and learn by using the journal as a tool, than by doing nothing until the end of term.

As a masterful procrastinator, the “end-of-term journal problem” is one I think about often. How can a course in which students must produce a sustained and reflective project be structured to best enable their success?

One solution is to require regular check-ins to ensure progress is made. However, if the project is meant to be private and reflective, weekly in-person checks are drastically inefficient for an instructor, even in smaller classes.

There is another way to check progress and provide feedback: move the project online using an eportfolio. eportfolios allow students to retain their privacy while granting the instructor access to check progress and leave comments about their work. Of course, a student might still fall behind, but I would have landed 13 pages closer to completing my visual journal had I known my prof would be checking my journal regularly, prior to the deadline.

The University of Saskatchewan eportfolio tool is our own version of open sourced Mahara. It provides a tool for students to collect, reflect, and share (if they choose) their work from one or more courses. Unlike Blackboard, a student’s eportfolio remains with them between courses and they are able to customize it based on course or even program requirements. It can store and display videos, photos, documents, and text. The layout can be customized, and it has features for planning tasks and writing reflective journal entries.

In the case of my studio course, for instance, I could have used an eportfolio to post articles I was reading in my Art History classes that were informing the art I made in the studio. I could have tracked and reflected on my progress by uploading photos of each piece in different stages of completion. The possibilities are numerous, and the ability for instructors to check-in quickly on students’ work, online and from anywhere at anytime, builds in a layer of accountability and support. This layer can help instructors track students’ learning at more points in time than only due dates and exams, and help students stay engaged with ongoing projects.