Me and Program Evaluation




I genuinely enjoy working as a Program Evaluator because the idea of efficiency and effectiveness genuinely appeal to me. In addition, being an applied and use-oriented person, the ability to use all my theoretical knowledge to help others is extremely fulfilling and appealing.

Program evaluation allows for effective resource allocation, documentation of need, improvement of effectiveness and efficiency, test novel small-scale interventions, address political issues and accountability. The different types of program evaluation are mainly about determining whether or not a program is needed, or being implemented as intended or achieving intended outcomes.

Needs in the context of program evaluation refers to the needs of the clients or participants of a program. There are times when we, as program developers or deliverers believe we understand the needs of a specific group, but in truth we do not. It is therefore important to verify the actual needs of the group.

Implementation has two aspects in program evaluation; the actual implementation and the intended implementation. This is very important in a program because the intended implementation is usually a theoretical one while the actual implementation is a function of the environment, participants, delivery and several other features. It is important to document the actual implementation, to ensure it is being implemented.

Program evaluation is not focused on goal attainment. It is ensuring that the program is in a state to achieve intended goals. Another important aspect of program evaluation and goal attainment is whether or not it is achieving unintended outcomes. Unintended outcomes can have positive or negative effects on participants or the community or other stakeholders.

As a result, program evaluation requires the awareness that all programs are situated within a context, and acknowledgment that the context affects, and is affected by, the existence of this program.

The concept of program evaluation is a simple one, however, in my experience it has required a shift in how I view the world in order to understand and apply the concepts of program evaluation. It also allows you to work in a variety of areas, which appeals to me on a very personal level, as I enjoy learning.

Program evaluation uses many of the same skills as research, but unlike basic research, it is applied; it is adaptable to the realities of working with people in their vast and complex world. It is all about serving the needs of the program and less about being a perfect piece of research. Also, by serving the program, it ends up serving the clients of the program, which, in my opinion, is extremely fulfilling.

Beauty II: Defining the Big, Bold and Beautiful




Getting lost in the beauty of our discipline and sharing it with our students raises the challenge of what to cover within the limited time of our course or program. With all that is beautiful about our discipline, what do we focus on?

32/365One approach is to focus on the fundamental perspectives and approaches that define a discipline – the building blocks of a field!

These building blocks can be identified and prioritized through several lenses:

Celebrating what’s Unique:

What makes your discipline unique? What are the key premises, approaches, conceptions, or methodologies not found in other disciplines? What is the unique contribution that individuals in your discipline can make to understanding human experience, and global and local challenges?

Defining Threshold Concepts:

One approach for defining these key ideas is by identifying the threshold concepts which are the transformative, troublesome, irreversible, integrative, and bounded concepts in a discipline that shape the language of the discipline (Meyer & Land). For example, Art as language in Art History (pages 7-9, http://www.usask.ca/gmcte/sites/default/files/Bridges_Aug_2012.pdf)

Resources:

Distilling the Essential:

Another approach is to distill from our activities and assessments what is truly essential, such as knowing the systems in anatomy versus naming the location of placement flags. Resources related to universal instructional design and accommodation for accessibility provide a starting place with open-ended questions for identifying the essential requirements or components in your own course or program:

Capturing in learning outcomes:

These big, bold, and beautiful pieces are the knowledge, skills and values at the foundation of each discipline and when articulated can form the program learning outcomes that define your curriculum vision.

The beauty that defines your discipline then becomes what that graduates of your program, those future builders of your discipline should know, do, value by the time they graduate. So that, by the end of this program/course, successful graduates are expected to…think in, act with and value with these big, bold and beautiful cornerstones of our discipline.

Being More Efficient



“efficient |iˈfiSHənt|

adjective

  • (esp. of a system or machine) achieving maximum productivity with minimum wasted effort or expense
  • (of a person) working in a well-organized and competent way
  • [ in combination ] preventing the wasteful use of a particular resource”   –(New Oxford American Dictionary”, 3rd Edition, 2010, Oxford University Press)

Efficiency focuses on the level of relevant output achieved relative to the amount of effort. Perhaps it is like the phrase “Work better, not harder” or “lift smarter, not harder”.

So what does being efficient mean for teaching? What does being efficient mean for curriculum renewal?

Set your goals:

Sailor switches his lights to high efficient bulbs as part of an energy conservation initiative in HawaiiFocus energy on the questions/areas that are most important.  Get the “most bang for your buck”. What is most important for your course or program may be different from others’ courses or programs, so being “efficient” may look different too. For example, how many people does it take to screw in a light bulb efficiently? One, if the goal is to disrupt the routine of as few people as possible; two, if safety is a priority and a ladder is involved; a whole class, if hands-on experience is valued.

Be selective:

You could measure everything all the time, or just what is sufficient to draw conclusions.  Choose the best measures for what you are focusing on. Try to assess important knowledge and skills to capture the signal more than noise.

Utilize the wheel instead of reinventing it:

Have an idea, but not sure of the detail? Look for what has worked for other people in your department, discipline, and beyond – there is a whole field of study on teaching and curriculum development! Unsure where to look? Ask a mentor, ask a colleague teaching a similar course or using the technique, or ask one of us at the GMCTE.

Think longitudinally/Avoid perfectionism:

Yes, this is the first time, but will likely not be the only time you will teach this topic or measure in this area. It won’t be perfect, but once it is good enough to try, try and then revise.

Document progress:

Sometimes it is hard to see how far we have travelled when we are looking at the next step and vaguely remembering the last one. Document the changes, the lessons learned and your successes. Share these insights with your students, colleagues, conference attendees and others. Allow students to see that you are building on their feedback, allow colleagues near and far to learn from your experiences and findings. If you’re thinking about presenting or publishing student responses and data, make sure to chat with the ethics office to find out about ethics exemptions and approvals.

The challenge of competing demands and the pressures to avoid wasting time and other valuable resources might seem a threat to good teaching, but perhaps they are a motivation for change. Similarly, following a schedule may be a constraint or a great way to Carve out Creating time.  I would love to hear how you have become more efficient in your teaching, or discuss quick ways of marking, surveying your program, or other intriguing challenges.  Our door is open at the GMCTE across from the Murray Library entrance – stop by on your way for coffee or drop us a line.

Picture courtesy of Official U.S. Navy Page via Flickr with a Creative Commons license (Attribution – Some rights reserved)

The Academic Honesty Bonus: Another Advantage of an Aligned Curriculum



In my role as a Curriculum Development Specialist, I get to talk with faculty about their programs and the many reasons to examine and renew curricula in higher education.  In recent months, another advantage to an aligned curriculum has come to mind for me:  academic honesty.

I posit that the three following relationships hold generally true, and promote academic honesty among students.

  1. When faculty alert students to the progressive nature of the curriculum and convey to students how what they are learning now prepares them for, not only life after graduation but for future courses, students can better recognize the benefit of deep learning.   For example, students can come to understand that the course on legal foundations of the profession in 2nd year prepares them for for the policy paper they will need to write as part of their capstone course in 4th year.  Or, students can learn that locating and reviewing journal articles for the intro course helps them learn how to write a research paper in the advanced course.   Naming such connections can provide just the “learning hook” needed to focus students’ attention and make them less likely to cheat themselves via academic dishonesty.
  2. When faculty relieve unnecessary time pressures on students, students become less likely to use cheating as a coping mechanism.  I acknowledge here that working well under time pressure can be a skill we wish students to learn, but in the case it is not a priority learning outcome it can be easily avoided.  For example, if faculty of the three required courses can time their class projects to be due in three different weeks rather than all in the same week, students will be in a better position to devote the time required for higher quality work.   With less unnecessary time pressure students also become less likely to resort to cut-and-paste “shortcuts” or full-blown plagiarism.
  3. When faculty know what assessments their colleagues use, they can be sure to build on past assessment practices and ensure they are not asking students to do the very same thing they have done for other courses.  Repetitive assessments can tempt students to submit work done previously for another class.  For example, the student who has returned from practicum experience and is asked to write reflection after reflection for each post-practicum course the following term, can come to the conclusion that the faculty don’t talk to each other about their teaching and therefore would never discover the “re-submission” or, perhaps worse, believe that the faculty actually doesn’t care.

And, here’s where curriculum conversations win the day! Fortunately, it is typical in curriculum renewal conversations that faculty have the opportunity to learn about what their colleagues teach, the learning outcomes they aim for, the assessments they use, and when they assess.   Given such conversations, most groups will then proceed to align their teaching for optimal learning conditions, and get the added bonus of having created the conditions that support academic honesty.

Is the Unexamined Program Really Worth Offering?


As we are being invited to take a vigorous look at the programs we are offering, I can’t help but wonder, haven’t we always been doing that? I mean, really, in this information age with new perspectives and burgeoning bags of “what we know” bursting at the seams on every possible topic, can we actually NOT be refreshing our program content annually at the very least? What was known last month is different than what we know this month! “Truth” is being regularly being rediscovered. Do you remember when the brontosaurus went the way of the dinosaur (so to speak) to be replaced by the new “truth” of the Apatosaurus or when we lost a planet or found another one?

Perspectives are shifting as we look around in our world and come to see that there are so many ways of knowing and interpreting that which we experience—not only do we have rose colored glasses but we also have fuchsia, chartreuse, pumpkin, and aquamarine! Given that we are from everywhere as a University community and going everywhere, how can we NOT be incorporating what we learn from our international colleagues (and I include students in “colleagues”)?

We have more information now than ever before about how our main interfacing tool, the brain, works. We can’t possibly be NOT adjusting how we teach given what we now know about how the brain learns, sorts, and bins incoming information.

This call to examine our programs is a huge opportunity to refresh, renew, and reinvigorate what we offer our students and our community. We don’t want to be passing along this tired old chestnut in the name of traditional, “it was good enough for me” when we know our programs could be even stronger, more relevant, and more engaging.

We all know what it feels like to be fired up and enthused about a new concept or being introduced to another way to view the world. We have all experienced the thrill of the ah-ha moment. We know what it feels like to be excited about learning and discovery.

Taking a look at your curriculum is like peering into a closet or pantry with curiosity and neutrality rather than with nostalgia, sentiment or apathy. “I wonder why we are doing this this way?” “Is this the best way?” “Do we still need this?”  “When was the last time anyone used this?” “Are we excited about our program and teaching?”—because if you aren’t no one else is going to be! Listen to new-comers who ask those questions rather than silencing them or telling them “this is how we do things here” when they ask “why.”

Embrace the chance to take your research lens to your curriculum with curiosity, inquisitiveness, neutrality, and genuine interest in what you might find. Be rigorous and thorough. Really want to know what the experience of your students is. Want to be the best you can be with what you’ve got. And then reach…

What is CAT 1.0?



So what is “CAT” beyond our nickname for our Curriculum Alignment Tool? It is an online software for academic programs and instructors to fill in information about your courses, connect those courses within a program, and review what you are teaching your students.

Designed as one approach to gathering data about programs, CAT fits into the inventory stage of the curriculum renewal cycle (see Susan & Sheryl’s dynamic video or diagram ).  Focusing on an instructor’s approach to their course, CAT asks about instructional strategies, assessment methods and timing, course outcomes and connection with program outcomes. CAT is particularly good at displaying trends and allowing for instructors to link parts of their course including outcomes and assessments.

Why use CAT? For instructors, this tool is designed to provide an opportunity for articulating their teaching strategies and goals, showing links between their goals and how they assess and meet program goals, reflecting on their strategies, and contributing information about their courses to their program curriculum renewal process.

From the prospective of programs, CAT provides an existing online tool designed for curriculum inventory that easily summarizes strategies and assessments across selected courses, shows relationship between courses and the program outcomes, and exports a more detailed set of data for further analysis. Depending on the goals of the curriculum renewal process, data can be entered in for a single year or tracked longitudinally when exploring change across offerings is desired.

For more information and resources see the CAT Website.

http://youtu.be/JhAnwm6WwqE

Developing CAT 1.0



When I first arrived at the GMCTE one of the first curriculum development projects I got involved in was the curriculum inventory tool. Initially called Currimap, it was in its initial feedback and trial stages and still growing: over the next few months, feedback from colleagues and faculty led to additional capabilities, refinements and flexibility being built into the code by our programmer. This fall we were pleased to launch our Curriculum Alignment Tool (CAT) 1.0. CAT is now an open-source resource available for those on our campus, and also with the code available for other institutions.
Throughout CAT’s development we have strived to balance (1) a straightforward interface and complex data management, (2) flexibility of input and complexity of analysis, and (3) functionality and protection of programs’ data. Our journey in creating CAT involved deep consideration of the purpose of CAT 1.0, based on the goals of programs we work with and our own knowledge as educational developers.

Balanced Rocks
Our goal was to create a straightforward system for describing courses, linking between components of each course’s design such as outcomes and assessments, and then connecting these courses together for a bigger picture of a program. By highlighting the features of courses, we intend to provide food for thought, reflection and consideration so instructors and program curriculum leaders can continue learning about what and how they are teaching.  Such data provides one piece within a broader data set for programs conducting an inventory as part of curriculum renewal (see Susan & Sheryl’s dynamic video or diagram about the cycle).

The challenge was to balance flexibility of input and complexity of analysis when deciding what components will be open for instructors and program administrators to change and what lists and layouts will be standardized.  The decision is similar to survey design where open-ended questions provide new insights but also require in-depth coding or narrative analysis, while closed-ended choices are quicker to summarize. We chose to provide some fixed lists along with areas for comments, with selections based on extensive feedback.

When data is being collected there is always the question of who has access and what the data will be used for. Developing CAT involved discussing and drafting a statement of data use and access to clearly outline what we will and will not do with the data. All data collected via CAT is for the use of the specific academic department or college who is asking their instructors to enter information. For functionality of CAT, data within a course or  program are used to populate lists for linking information previously entered such as assessments or outcomes. As educational developers we also have access to answer questions, select options or otherwise assist programs. When sharing information about CAT, we will be only able to share aggregated data, our fictional demo program, or any programs that volunteer. Full details of the use and access policy are in the guide

For more information and resources about CAT visit http://www.usask.ca/gmcte/CAT