Practice Problem Sets: Issues of Timing and Mixing

While looking for resources for a faculty member in the sciences who was interested in incorporating more problem sets into her lectures to increase student engagement, I came upon a 2007 article by Rohere and Taylor, appearing in Instructional Science. This article describes two experiments where particular timing and mixing of mathematics practice problems improved learning.

The authors point out that it is usual for practice problems to be assigned:
• immediately following the relevant lesson (massed), and
• for problems of the same type to be grouped together (blocked).

15 - September - 2008 -- MathsThrough Rohere and Taylor’s experiments, they found that spacing the timing of two sets of practice problems 1 week apart (they called this spaced rather than massed) and varying the types of problems in a practice set (they called this mixed rather than blocked) greatly improved students’ test performance.

While the experiments used math concepts (one was a permutation task, the other was a volume task), it seems there could be an extrapolation/application to other kinds of practice problem sets for students.

The basic idea I take from this article for teaching is twofold:
(1) have students return to the problem type practiced in the previous week, and
(2) mix last week’s problem type with this week’s problem type.

This approach means students get to try a set of problems again—important especially if they had difficulty first time around. Plus, using mixtures of problems handled at the same time requires students to learn to pair each kind of problem with the appropriate procedure – that is students not only learn how to perform each procedure (learning-how), but also which procedure is for each kind of problem (learning-which) – the authors call this “discrimination training.”

A powerful closing remark in the article is that shuffling problems in this way presented few logistical demands for the teacher, making it an easy change in teaching practice that can have dramatic benefits for student learning.

(And, here I’ll add another easy change that builds on the above…teachers can ask students to do problems sets in small groups, then exchange them with another group and provide peer feedback on their calculations or choice of procedures. The learning that occurs by providing feedback to peers has also shown improvements in student performance.)

Rohere, D, & Taylor, K. (2007). The shuffling of mathematics problems improves learning. Instructional Science, 35, 381-498.

Evaluating Presentations With a Little Help From My (Citable) Friends …

Individual and group presentations provide great opportunity for students to share what they have learned with peers and an efficient and feasible way of marking for instructors.

That being said, how do you grade them?

I, and I’m pretty sure you too, have experienced the full range of presentations from the stunningly excellent to the staggeringly confusing, from the inspirational to the sleep-inducing. The challenge is describing these qualities so they can be identified and assessed.

One option would be to create my own rubric based on these experiences.

The easier option is to use or adapt existing materials from others I respect.

The first source I turn to is the well-respected Association of American Colleges & Universities’ VALUE (Valid Assessment of Learning in Undergraduate Education) assessment initiative, which has created 16 rubrics including one for oral communication.

They define oral communication as “a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners’ attitudes, values, beliefs, or behaviors” and assess it according to five criteria: organization, language, delivery, supporting material and central message. The rubric describes requirements for each criterion across 4 levels. For example, Capstone (4) level delivery requires that presenters’ “Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation compelling, and speaker appears polished and confident.”

The second resource I consider is the more detailed (15 criteria across 3 categories) rubric of the American Evaluation Association’s Potent Presentation Initiative (p2i). Their website has every resource I ever wished to send to students (and perhaps others) about what “good” presentation or posters look like. They have posted rubrics, guidelines, templates and resources for regular slides presentations, ignite presentations (20 slides x 15 seconds = 5 minutes of auto-advancing slides), posters and handouts.

In addition I could ask a colleague or see what other courses in the program are using.

presentation outline

One of the best parts of adapting rubrics is the opportunity to decide which pieces I find most important for my course (e.g., organization), ones that are relevant if revised to be more specific (e.g., supporting material) and ones that are not (e.g., mastery – speaking without reading from notes). I also can decide which resources I recommend (e.g., p2i slide design guidelines), which I comment on (e.g., I suggest noting times if you use the p2i rundown template) and which I just mention (e.g., p2i presentation preparation checklist).

When uncertain I can always ask for a second opinion from a colleague, request a consultation, or trial it before posting the criteria.

Happy assessing!

Picture courtesy of Sean MacEntee and carries a Creative Commons Attribution license.

Undergraduate Research: Co-Publishing With Students

By Jason Perepelkin, Assistant Professor, College of Pharmacy and Nutrition

Passive listening and dumping information on exams doesn’t give students the depth of learning and experience that lasts beyond the scope of a course. Having students engage with practitioners and specialists and in a real world environment helps students learn more deeply; chasing grades doesn’t do this but chasing experience does.

The elective fourth year course Marketing for Pharmacists is designed for up to 20 students. The course is a project based course where students, working in groups of two to three, work directly with a practicing pharmacist. By working directly with practitioners, on an issue identified by the practitioner, the students learn, in a hands-on manner, about a specific practice site, while the practitioner learns about marketing and how it can be used to enhance practice.

In the first year the course was offered there were 20 students, which is the maximum. This year, 6 students were enrolled, and as a result it could be run much more as a seminar. Half way through the course I thought (out loud), based on the enriching discussions around current events in pharmacy, if I was thinking we would’ve written a manuscript on these issues. The students came back a week later and said “can we do this?” I said only if all of you are willing to be involved. They said yes, so I approached a journal to see if they would be interested in an article surrounding our class discussions; the journal responded indicating their interest.

After working on the article as a group, and in consultation with myself, we submitted the manuscript for peer-review to the Canadian Pharmacists Journal at the beginning of December. In early January we received notification that our manuscript was accepted for publication, but required some minor revisions first. Since the students were not in the course anymore, and were out on experiential learning rotations across the country, I wasn’t sure if I was the one that would be completing the revisions; however, the students jumped at the chance to revise the manuscript, and even spoke of how they learnt, from the reviewers, how the manuscript can be enhanced. This allowed the students to experience the entire process, from the idea, to the research and drafting of the manuscript, to receiving feedback from peer-reviewers, and ultimately to acceptance. The manuscript was accepted the day after the revised manuscript was resubmitted, and will be published in the May/June 2015 issue.

I am not sure if this would work as well as it did, especially since it arose – after the course was half completed – from an organic process of critical thinking and discussion in class, with a different group because the maturity of the group and their willingness to cooperate was very high. As a sign of maturity, at the beginning of the course when students are to form groups of their choice, all agreed they were willing to work with anyone in the course (despite not being in the same ‘clicks’), and therefore I put all of their names in a hat and randomly selected members of each group.

Some students want to do this sort of a project and these students are the ones working on projects before they even start the course. If enrollment increased, it would be harder to ensure all papers got published and this could lead to disappointment for the students. A smaller class allows full participation in the publishing process, and in the course as a whole.

Context is incredibly important in making this work. For some students in this college marks are not as important as experience and peer-accountability is in motivating them to first enroll in the course, and second engage in the course and project. This sort of course gives students a different experience from traditional pharmacy courses, and brings recognition to other concerns such as how marketing can be used to better meet the needs of patients and the health care system as a whole. This is the first course of its kind in Canada, and provides those students that take the course the ability to learn a unique skill set that is not readily available once they enter practice; there are only a minimal number of continuing education opportunities in the area of marketing.

Developing ePublications

By Adrienne Thomas and Wayne Giesbrecht (Media Production)

With discussion surrounding open resources, this is a good time to talk about actually developing epublications and ebooks. For the past 3 years, Media Production (formerly eMAP) has been working with faculty and content creators to realize epub resources. With each new project, we have learned more about what to do and how to do it – an ongoing lesson as the software, media files and platforms continue to evolve.

Interprofessional Skills Learning GuideWithin the university environment, we are all concerned with the development of unique and immersive material to be used for information, education, research or knowledge mobilization purposes. If you want to make your content available as an epublication, you need to first determine who your end users are, and secondly, how they will access the material. Once you have made these decisions, it is a matter of formatting your content and designing a publication which will meet your informational or educational objectives to greatest effect. This can be a relatively easy process such as a PDF document converted to an epub format for web browser access to a more involved publication with media-rich/interactive content to be distributed across multiple supported platforms.

When thinking about access for your readers, you will also need to determine if this material will be open or free, or if it will be monetized and distributed commercially. If you self-publish, there is an opportunity for not only creative control, but price control. This was a major consideration for Dr. Bruce Grahn when he decided to e-publish his last textbook Ocular Diseases of Companion Animals for international distribution. Working with Dr. Grahn and the associated contributors, we formatted a full reflow etextbook, navigated account setups, acquired an ISBN and the required approvals (with associated proprietary file formats!) from commercial distributors. The textbook is now available for purchase on ITunes and Google Play.

The creation of the text was cost effective and any future revisions will automatically be updated in all distributed editions at no extra cost to the end user.

A media-rich experience and end user access were the requirements when we started working on an interprofessional education guide for the College of Medicine. There was a need for flexible access which would allow for independent learning or small discussion groups via mobile devices. Working with Heather Ward and Dylan Chipperfield, this project allowed us to develop an ebook which used video to moderate the content and present simulations. Embedding video within the epub presented interesting challenges, particularly for multi-platform access. When adding media-rich content (video, audio, animations, quizzes etc.), file size, reflow and platform incompatibility can be problematic and requires more consideration in layout and formatting. The project, Interprofessional Skills Learning Guide, was completed and is now accessed by health care professionals through the College of Medicine website, on ITunes and Google Play.

`Traditional publishers are invested in ebooks, it is an emerging technology likely to hold. It is also a gateway to open source educational materials and immersive experiences for students.

For more information about creating an eBook, please contact Adrienne at (306) 966-4280 or Wayne at (306) 966-4287.

What is Digital Citizenship?

Many teaching and learning conversations include notions of developing and fostering citizenship for our teachers and our learners in our respective disciplines and fields and in society.   Citizenship can be such broad territory. One way to focus it further is to discuss Digital Citizenship. If you’re still stumped, let me point you to a useful set of Nine Elements of Digital Citizenship appearing on a web site dedicated to this topic. Here, among other things, you’ll find types of norms that characterize appropriate and response technology use.

The distinctions between digital literacy, digital communication, digital etiquette, and digital rights and responsibilities strike me as most informative. When we exchange information electronically with others, we are engaging in digital communication.   When we are in the process of learning about technology and how to use technology to learn, we are becoming digitally literate. When we adhere to standards of conduct or procedures in our use of technology, we are upholding digital etiquette. When we accept our rights to privacy and free speech as well as our responsibility to use technology appropriately, we are recognizing digital rights and responsibilities.

Given recent concerns over offensive behavior on Facebook by a group of Canadian university students, additional digital citizenship notions of protecting confidential communication from being breached (digital security or self-protection) and taking responsibility of actions and deeds (digital law) also bring insight.

If you’re like me, looking for some clarifying definitions to assist your role and the roles of others in digital citizenship, you’ll find this useful too. Check out the rest of the website and learn more about Mike Ribble.

What It Means to Be an Ally

As we have recently come out of a week of sessions at the University aimed at making our campus a safer place for gender and sexual diversity and we enter Aboriginal Achievement week I am reflecting on what it means to me to be an ally.

Use of the term ‘ally’ in relation to marginalized groups is relatively new to me, however, what the term represents is not new.
Being an ally means working in solidarity with a marginalized group that I am not a part of to address systemic inequalities.

I’ve tried to boil down what I feel I have to work at everyday in being an ally (some days more successfully than others!) and have come up with 5 key things I’d like to share:

1)   I have to understand my position of privilege
This privilege is something that I have not earned, but received simply because of my characteristics – the way I was born.

When describing this to my six year old daughter, I liken this to recognizing that some of us are playing this game on level 1, while others are on level 5.
There are fewer barriers for me, fewer obstacles in my way and its far easier for me to get to the finish line. This doesn’t mean I haven’t worked hard or that I’ve sailed through life or not met challenges.  It just means there are things I don’t have to worry about ever facing because of who I am.

I don’t have to worry about what bathroom I may use in the mall or at work because they have been designated female and male with me in mind.
I don’t have to worry about a job application I put in being set aside simply because of the way my name sounds. I also don’t have to worry about being watched while browsing in a store simply because I am less likely to be viewed as suspicious because of the colour of my skin.

I’ve never had to get through the game at level 5, but it is my job as an ally to find out what its like as best I can, acknowledge and accept that some things are easier for me, and take what action I can to contribute to levelling the playing field.

2)   I need to listen and learn.
I need to work to listen to concerns raised by marginalized groups.  I need to consider them thoughtfully and recognize that at times my position of privilege can mean experiences sound unbelievable – they are so removed from my reality. Listening receptively can mean a marginalized group and the barrier they face can become more visible.

3)   I need to consider my position in making change.
As someone in the dominant group I should not be at the centre of the solution, I should only be a part of it.  This means dropping my agenda and my way of change. The marginalized community should be at the centre and I should be there to do what I can to contribute to making it happen.

4)   I need to accept that I will mess up and be uncomfortable and that I just have to deal with that.
Being a farm girl, I liken this to crossing a cow filled pasture.  If I focus ahead with my eyes on the horizon, I am going to step in it on occasion.  When it happens I need to apologise, learn from it, bend down, clean off my boots and keep going.

5)   Last but actually most importantly I need to be aware that being an ally is a daily activity I wake up and commit to doing – not a title or certificate I earn.
It is a verb not a noun.

So as I move forward in my work as an ally for the LGBT community, the Aboriginal community, or other marginalized groups I will work:

  1. to recognize my privilege;
  2. to listen;
  3. to find my appropriate place in driving change;
  4. accept I will mess up and I should learn from it; and
  5. keep trying

If you notice me step in something, I welcome you bring it to my attention so I can apologize, clean off my boots and continue to learn.

In writing this I read several sources. I’d recommend this blog post if you would like to read more or if you’re short on time, this 3 minute video is engaging and concise.  It might be a good one to share with students.

How to Jump Start You Curriculum Innovation

Curriculum and teaching innovation are rarely held back, ultimately, by money alone. But, if getting a bit of money could get your initiative moving ahead, you might like some help from the Curriculum Innovation Fund.

What is the Curriculum Innovation Fund?
The CI Fund is intended to provide financial support to curriculum innovation and renewal projects at the University of Saskatchewan that specifically change or develop content or methods of a collection of courses.   The fund can support initiatives focused on a single course that pilot an innovation or show potential to contribute to program-level change.

Why was the Curriculum Innovation Fund established?
The fund was established by the Provost’s Committee on Integrated Planning (PCIP) to implement the priorities of IP3 and specifically to support innovation in academic programs and services. The fund is allocated $250K annually, beginning in 2012-2013 and ending in 2016-2017. For a list of projects and recipients to date, see:

What is the application process?
The application is in the form of a 4-page word document with word limits suggested, point form accepted, and a budget table provided. Drafting assistance is available from the staff at the GMCTE. Applications are accepted at any time and response usually occurs within 1-3 weeks.

How to start?
Contact us at the GMCTE, check out our comprehensive website, and consider coming to a workshop for hands on drafting assistance on Feburary 4 or March 4. Click here for more information and to register.

Connecting Ideas for Innovation

Connecting Ideas for Innovation

When do your great ideas come to you? Where do your great ideas come to you? Is it when you’re alone in your office or lab? How about when you’re out for a walk?<

Are you sure about that?

Steven Johnson, the author of How We Got to Now, Everything Bad is Good for You, and Where Good Ideas Come From argues that while bits and pieces of those ideas may come together in your solitude, they actually become really good or even great ideas when they have a chance to mingle with other ideas.

In his TED Talk Where Good Ideas Come From (see the video below), Johnson argues that the really great ideas come when a percolating idea from one person encounters those of other individuals. It happens at meetings with colleagues, those in-between conference session conversations, and, in his main example, places like coffee houses (he notes the original coffee houses of London to start the point rolling).

Johnson gets to the heart of this his Ideas book:

“But the truth is, when one looks at innovation in nature and in culture, environments that build walls around good ideas tend to be less innovative in the long run than more open-ended environments. Good ideas may not want to be free, but they do want to connect, fuse, recombine. They want to reinvent themselves by crossing conceptual borders. They want to complete each other as much as they want to compete.” (Johnson, 2010, p. 22)

We need to help our ideas connect. We need to put ourselves in those situations where serendipity can happen.

Universities are prone to siloing. A siloed structure encourages people stay within their colleges, sometimes just within their departments and this can be a problem if we want to encourage creativity. I’ve seen early forms of ideas blossom when a faculty member from one department has a conversation with a colleague from across campus at our Course Design Institute around something as basic as assessment. I’ve seen blog posts about one concept lead to ongoing conversations across the Web leading to new courses or research topics. If we remain within our silos or cloistered away in our offices with no flow of ideas outside of our heads, we could be doing ourselves and our ideas a great disservice.

As Johnson said, ideas want to connect, they want to evolve with the input of others, they want to join forces with the ideas of others and become great.

Johnson, S. (2010) Where Good Ideas Come From: The natural history of innovation.  New York: Riverhead Books.

Authentic Assessment

I think of authentic assessments as ways for students to demonstrate knowledge and understanding in a public way. What makes assessment authentic for me is that students do something to show what they know in a public way that benefits a wider community than the one person assigning a grade.

The posters that students did in their first year College of Agriculture and Bioresources (AgBio) classes this past term are, in my way of defining authentic assessment, stellar examples.

Working in teams, students prepared a research poster as part of their undergraduate research experience. On the afternoon of December 3rd there were 99 posters on display up and down hallways in AgBio. What an impressive and exciting initiative!

I spent the afternoon asking students about their research and reading posters. I spent time with about 30 of the 99 teams. All students were articulate, knowledgeable, engaged, and prepared. In comparing notes with the other judges, this was the rule—without exception.

It was a great afternoon—and example authentic assessment—students got direct feedback through their interaction with their peers, instructors, and people walking through AgBio. They received feedback from their instructors throughout the research process that culminated in the poster afternoon.

It was a big undertaking for the students, instructors, TAs, and research coaches involved but the impact, the connections, the sharing of ideas and information, the chance to discuss and explain…It was powerful. And I think it created memories that will not be forgotten for a long time—if ever.

So the students could have handed in an individual report on a question they were interested in and the instructors could have been the only people who benefitted from reading what the students wrote, but this was an event.

If you have an example of when you have implemented an authentic assessment strategy in your class or you are interested in brainstorming and planning ways to integrate authentic assessment in your class please contact us at the Gwenna Moss Centre.

Note: AgBio is one of the three pilot sites and the instructors of the first year courses took up the initiative in a big way! Their goal was for every AgBio student to have an undergraduate research experience in the first term of their first year. (For more on this initiative check out a couple of Murray Drew’s blog posts:


Mental Illness, Disability, and the Inclusive Classroom

By Adam Pottle, Graduate Fellow

In its Campus Climate survey report, which was released in November 2014, the University of Saskatchewan identified a number of areas it needs to improve in terms of making students feel safe and comfortable. The survey summary, which can be found at, reports that

 [s]ome students in minority groups had less positive experiences when compared to their counterparts, especially some Aboriginal students, other visible minority students, sexual minority students, and some students with a disability. On average, those indicating a mental health condition generally had fewer positive experiences than all other students. (4)

The survey goes on to state that “57% of those with a mental health condition reported they either considered leaving or did leave the U of S” and that students “with a mental health condition…were twice as likely to report experiencing insensitive behaviour, exclusion, harassment and/or discrimination as compared to the overall survey population” (4). These results are troubling, especially given recent nationwide efforts to generate awareness and discussion, such as Bell’s “Let’s Talk” program. Clearly, more work is needed.

Mental illness has long been a stigmatized condition, namely because it is difficult for people to understand. In the classroom, it is easier to understand and include students with physical disabilities because any hindrance to accessibility assumes a tangible, physical form. Spaces can be rearranged; lectures can be recorded; hearing devices can be implemented. Mental illnesses such as schizophrenia, depression, anxiety, and paranoia cannot be seen with the naked eye. They are obscure, and because we tend to fear what we do not understand, we stigmatize these conditions.

To help provoke dialogue around this subject, and to help instructors devise teaching strategies to promote accessibility, the Gwenna Moss Centre has created a workshop called “The Inclusive Classroom: Fostering Accessibility for Students with Disabilities.” The first offering of this workshop will be held on Monday February 9, 2015 from 1:30 to 4 PM and will feature presenters from Disability Services, the Canadian Mental Health Association, and the Learning Disabilities Association of Saskatchewan. Although the workshop will consider all disabilities, it will focus on mental illness and learning disabilities because these two conditions most commonly affect students at the University of Saskatchewan. Participants will discuss how to create open, inclusive classrooms and how to employ diverse teaching strategies for students with disabilities. This workshop will hopefully generate productive dialogue and help diminish the stigma of disability and mental illness.

The workshop will be held Murray 102. To register, please visit the events page on Gwenna Moss website and scroll down to February 9.