Overview

Oregon State University began migrating to Canvas in January 2015, and ran Blackboard in parallel for four academic terms (through December 2015) for all modes of instruction: face to face, online, and hybrid. Winter 2016 was the first term we used Canvas exclusively.

Near the end of spring term 2016 the OSU Canvas team distributed two surveys, one designed for faculty and staff, and another for students. Both surveys included questions to solicit quantitative and qualitative feedback on perceived value of Canvas tools and features as well as use of and satisfaction with Canvas support options. Instructors were also asked about their familiarity with OSU’s Teaching and Learning Expectations, a foundational set of guidelines generated by OSU faculty and students in 2012 upon which OSU’s subsequent LMS evaluation and selection process was based.

This document provides a high-level summary of the results of both surveys, including features with which users are particularly dissatisfied or unfamiliar.

Survey Design

The surveys were drafted by Lynn Greenough (Canvas product manager in Academic Technology) with guidance from Katie Linder, Research Director for Extended Campus; Katy Williams, instructor of Mathematics and Statistics, and Cub Kahn, Center for Teaching and Learning.

The student and instructor surveys were both created in Qualtrics. The instructor survey was sent out to several Inform email lists and also to a 160-member email list that anyone may subscribe to for updates about Canvas at OSU. A link to the student survey was posted on the MyOSU student portal, and Lynn also requested department chairs to send the survey to their student email lists.

Result Summary

Faculty

The top areas of satisfaction with Canvas are:

  • File Upload and Availability (78-85% satisfaction)
  • Assignment Submission Features (68-79% satisfaction)
  • Linking to Course Materials (73% satisfaction)

The top areas of dissatisfaction are:

  • Canvas Conversations (16-17% dissatisfaction)
  • Assignment Grading Features (9-12% dissatisfaction)
  • Collaboration Tools (5% dissatisfaction)

Students

The top areas of satisfaction with Canvas are:

  • Assignment Submission Features (72-86% satisfaction)
  • Discussion Board Features (83% satisfaction)
  • Viewing Rubrics (80% satisfaction)

The top areas of dissatisfaction are:

  • The Canvas Mobile App (15% dissatisfaction)
  • Adding/Embedding Video (13% dissatisfaction)
  • Canvas Conversations (9-12% dissatisfaction)

Recommendations

Based on qualitative and quantitative feedback from students and instructors, the Learn@OregonState Advisory Committee recommends the following:

  1. Expand Academic Technology’s faculty training and engagement strategy.
    • Tailor support offerings to specific populations, particularly campus-based instructors.
    • Expand training and support on Canvas course-level analytics
      • OSU resources available? Yes
      • Level of effort: Medium
  2. Develop more course templates with optimal course design and layout.
    • Provide exemplary course models.
      • OSU resources available? Yes
      • Level of effort: Medium
  3. Develop tools for large-enrollment course management.
    • OSU resources available? No; this work requires a developer familiar with the Canvas programming environment.
    • Level of effort: Potentially high, pending analysis
  4. Develop and conduct regular reporting to quantify instructor use of Canvas tools and features
    • OSU resources available? Yes; technical report development work needs to be prioritized
    • Level of effort: Medium
  5. Offer Canvas course navigation support to students at the beginning of fall, winter and spring terms
    • OSU resources available? Yes; student workers in Academic Technology provide similar support for Turning clickers during these times
    • Level of effort: Medium
  6. Identify impact of low faculty awareness of OSU’s Teaching and Learning Expectations
    • OSU resources available? Yes
    • Level of effort: Low

Detailed Survey Results

Faculty Survey

225 instructors started the survey; 204 completed it, including some advisors and professional faculty. This represents approximately 10% of OSU instructional staff. The largest reported Canvas sections include 600-700 students. Many have 100-300.

Table 1. College Affiliation
Primary college or unit responders are affiliated with % Count
Liberal Arts 19.11% 43
Science 17.78% 40
Agricultural Sciences 13.33% 30
Public Health and Human Sciences 11.56% 26
Business 8.44% 19
Engineering 8.44% 19
Other 5.33% 12
Veterinary Medicine 4.00% 9
Earth, Ocean and Atmospheric Sciences 3.56% 8
INTO OSU 3.56% 8
Education 2.67% 6
Forestry 2.67% 6
Graduate School 2.67% 6
Student Affairs 2.22% 5
Academic Affairs 1.78% 4
Honors College 1.78% 4
Pharmacy 1.33% 3
Professional and Continuing Education 0.00% 0
Total 100% 225
Table 2. Course Type
Types of courses taught % Count
Campus-based 85.78% 193
Ecampus 38.67% 87
Hybrid 10.22% 23
N/A - I do not teach using Canvas 4.00% 9
Extension 0.89% 2
Professional and Continuing Education 0.44% 1
Total 100% 225
Table 3. Position Type
Position type % Count
Instructor / Sr. Instructor 44.89% 101
Faculty (tenured/tenure-track) 39.56% 89
Professional Faculty (non-advisor) 10.22% 23
Advisor 6.67% 15
Other 5.33% 12
Graduate Teaching Assistant 2.22% 5
Table 4. Faculty Dissatisfaction with Canvas Tools
Canvas tools with which instructors are most dissatisfied Very dissatisfied Dissatisfied Somewhat dissatisfied Total
Send an email to your students via the Canvas Inbox 12 13 10 35
Find emails your students send you from Canvas 13 14 5 32
Create an assignment so that you have a column in the gradebook to manually enter grades 8 8 8 24
Give your students feedback on an assignment 7 7 4 18
Table 5. Faculty Knowledge of Canvas Tools
Tools about which instructors ranked themselves as least knowledgeable Not at all knowledgeable Somewhat knowledgeable Total
Use Student View 32 47 79
Use the Calendar (enter a due date on an assignment so that it automatically populates the calendar and/or create an event so students can see details on the course calendar) 39 59 98
Create a rubric that you can use to grade an assignment 56 47 103
Adjust notification preferences 49 81 130
View course analytics 69 76 145

The concern about instructors’ lack of knowledge about these tools is that each of the tools has the potential to:

  • Help students find course material and know when it’s due (Calendar)
  • Understand how their work will be evaluated, and what the criteria are for learning mastery (Rubric)
  • Know what the course layout, materials, assignments, grades, and instructor feedback look like from the student point of view (Student View)
  • Quickly view student participation and identify potentially at-risk students (View Course Analytics)

There are a handful of Canvas tools with limited pedagogical use (Chat); that have another service already supported by OSU (Conferences); or that have some pedagogical use but are more difficult for faculty to implement because course members need to register Google accounts in Canvas (Collaborations). These tools rank higher on the ‘not knowledgeable’ scale, but the impact on student learning or course site quality is negligible.

Ecampus instructors

Among instructors who teach one or more Ecampus courses, their ratings of “very knowledgeable” are typically 10-30% higher than the campus average. Given that Ecampus instructors use the tools daily, these results make sense. However, there were still issues with knowledge of rubrics and Kaltura. The Chat, Conferences, and Collaborations tools have the exact same spread among Ecampus instructors as the full survey population.

Table 6. *Faculty Frequency Use of Support
Rank and frequency of the ways to get technical and instructional support for Canvas Mean (Min=1; Max=10) Standard Deviation Count
Canvas online Guides 6.18 2.77 155
Canvas tech support (phone; live chat; email) 5.45 3.06 134
General Web search 5.12 3.13 115
Ask a colleague 5.05 2.94 149
Ask Ecampus staff 4.40 3.25 90
Ask TAC or IT staff 4.24 3.02 105
Canvas Transition Web site 3.86 2.64 85
Ecampus Web site 3.53 2.80 85
Other (your fill in) 2.52 2.70 31

*This support frequency ranking question was only made visible to approximately 100 responders who indicated in a prior question that they use any form of Canvas support.

Table 7. Awareness of Teaching and Learning Expectations
Indicate your level of knowledge about OSU's Teaching and Learning Expectations % Count
Not at all knowledgeable 57.84% 118
Somewhat knowledgeable 26.47% 54
Knowledgeable 10.29% 21
Very knowledgeable 5.39% 11
Total 100% 204

The impact of low level of awareness of OSU’s Teaching and Learning Expectations is not known at this time. Some representative comments:

  • Should I know what this is? It's not ringing a bell with me.
  • I was a part of the evaluation team so that's why I know about these expectations....I don't think faculty in general paid much attention.
  • Where are these provided for instructors beyond the transition site link? How are they articulated and assessed?
Table 8. Overall Faculty Satisfaction with Canvas (18% dissatisfied; 70% satisfied)
Answer % Count
Very dissatisfied 3.92% 8
Dissatisfied 7.35% 15
Somewhat dissatisfied 6.86% 14
Neither satisfied nor dissatisfied 12.25% 25
Somewhat satisfied 21.08% 42
Satisfied 36.76% 75
Very satisfied 11.76% 24
Total 100% 204

The satisfaction levels with Canvas are identical between Ecampus instructors and the full instructor survey population.

Representative comments from instructors

Canvas has a rather steep learning curve which takes time that I do not have to become operational. As a result I use it at a very rudimentary level.

  • I'm sure Canvas is more powerful than Blackboard, and we can all be proud of this new whiz-bang masterpiece, but it's a real pain in the ass to change
  • I'm dissatisfied with the Canvas Gradebook which doesn't record grading history. I'm also dissatisfied with the upload process which doesn't allow changing the filename before uploading an Excel gradebook. I'm very dissatisfied with the Canvas messaging system (I don't consider it an email system). I'm dissatisfied with the fact that Canvas cannot easily split a Gradebook into sections and then remerge them to create a master gradebook. I'm also dissatisfied with the fact that sections of students cannot be messaged simultaneously without putting in a great amount of effort to create a site for each section. In other words you can't input a field into the Canvas gradebook titled "section number" and then filter by section number to message only those students. If you want to add an extra column to the gradebook you have to add an "assignment" which then has to have a grade associated with it.
  • Canvas generally works well. The online quiz tool is great. Multithreaded discussions, extra credit and TurnItIn need better implementation.
  • The grade center was difficult to use for a course with several instructors. I culd not figureout how to make section grades and then be able to weight the sections.
  • I am looking forward to spending more time using various features to enhance my class.
  • it's getting there - but there are still ways in which it makes our work harder and longer. Overall, it's an amazing tool and I'm glad we have it.
  • There are alot of things that Canvas doesn't do or is very rigid in how it does things. It can be quite frustrating. I also hate the fact that Canvas is so "open." I didn't realize that students could see a bunch of things that I didn't want them to see!!
  • I am puzzled why we moved away from Blackboard. These programs provide essentially same functionality.

Student Survey

501 students started the survey, and 360 completed it. This represents just over 1% of OSU students.

Table 9. Student College Affiliation
In which college is your primary major? % Count
Agricultural Sciences 26.35% 132
Liberal Arts 18.96% 95
Engineering 18.76% 94
Forestry 7.39% 37
Public Health and Human Sciences 7.39% 37
Earth, Ocean and Atmospheric Sciences 6.59% 33
Business 6.19% 31
Education 5.79% 29
Honors College 1.20% 6
University Exploratory Studies Program 1.00% 5
Pharmacy 0.20% 1
Veterinary Medicine 0.20% 1
Total 100% 501
Table 10. Student Class Standing
Class standing as of S2016 % Count
*95% of "other" responses are Post Bacc, along with a few graduate certificates
Senior 29.34% 147
Junior 26.35% 132
Other (your fill-in)* 18.76% 94
Sophomore 11.18% 56
Master's 7.19% 36
Freshman 4.79% 24
PhD / Ed.D / Pharm.D 2.40% 12
Total 100% 501
Table 11. Student Dissatisfaction with Canvas Tools
Tool Very dissatisfied Dissatisfied Somewhat dissatisfied Total
Canvas mobile app 14 20 19 53
Find content in your Canvas course: syllabus, assignments, lecture slides, readings, videos, etc. 15 20 13 48
Embed a video in an assignment or discussion 13 16 19 48
Send an email to your instructor from Canvas 16 11 20 47
Send your instructor comments on an assignment or a grade 13 14 16 43
Table 2. Student Support Effectiveness
Support Effectiveness Mean (Min=1; Max=10) Standard Deviation Count
Canvas tech support (phone; live chat; email) 6.96 3.36 55
Canvas online Guides 5.99 2.97 70
Ask my instructor 5.89 3.08 82
Ask a friend 5.61 2.97 59
General Web search 5.52 3.06 56
OSU computer helpdesk 5.49 3.18 43
Other (your fill in) 4.88 3.64 16
Table 13. *Student Satisfaction with Support (6% dissatisfaction; 46% satisfaction)
Answer % Count
*This question was only made visible to students who responded to the previous question about effectiveness of support options
Very dissatisfied 1.11% 4
Dissatisfied 3.61% 13
Somewhat dissatisfied 1.39% 5
Neither satisfied nor dissatisfied 47.50% 171
Somewhat satisfied 8.61% 31
Satisfied 26.39% 95
Very satisfied 11.39% 41
Total 100% 360
Table 14. Student Overall Satisfaction with Canvas (11% dissatisfaction; 84% satisfaction)
Answer % Count
Very dissatisfied 3.61% 13
Dissatisfied 3.89% 14
Somewhat dissatisfied 3.33% 12
Neither satisfied nor dissatisfied 5.56% 20
Somewhat satisfied 10.00% 36
Satisfied 42.50% 153
Very satisfied 31.11% 112
Total 100% 360
Representative comments from students
  • Teach instructors to use Canvas better and utilize it's tools
  • Overall it is very easy to use, just work on the app more please.
  • this isn't really negative to Canvas, but I would like to see instructors all use canvas in the same way. Some are very good, and some are really confusing. It would be fantastic if there was some quality control and uniformity on how the courses are set up in canvas.
  • Better notification settings would probably benefit everyone!
  • Something about the organization of the site seems disorganized and unintuitive. I can't put my finger on it and it may be due to how the instructor's select tabs and organization in their own course, but it feels disorganized and redundant and hard to find things.
  • Instructor organization of material makes a significant difference on results of the user friendly canvas experience
  • Improve the mobile app and discussion features

Conclusion

While the official learning management system migration work is behind us, survey responses indicate there are many ways that Oregon State University can support instructors and students to make better use of Canvas and its growing suite of integrated tools. The Learn@OregonState Advisory committee recognizes the high level of quality that Ecampus brings to fully-online courses. In fact, Ecampus resources and expertise have been a critical component in the successful transition to Canvas. We have identified six actions that can be undertaken to improve faculty use of Canvas and student experiences with Canvas. Most of the actions will be focused on campus-based instructors (all faculty ranks, including TA’s), but we are confident that improvements will benefit all modalities of teaching and learning at OSU.

  1. By expanding Academic Technology’s faculty training and engagement, we will be able to tailor support offerings to specific populations and address their unique challenges with Canvas. In particular, providing tailored support to the multitude of campus-based instructors. In addition, this training will raise awareness of Canvas analytics tools which will, at a minimum, improve course-level assessments, and potentially contribute to program and university learning outcome assessments as well.
  2. More easily-deployable course templates and course models will improve course design, layout and accessibility consistency across campus, minimize faculty time spent on these Canvas structures, and hopefully shift faculty focus to the development of pedagogical tools instead.
  3. We acknowledge the challenges encountered by faculty instructors of large-enrollment courses, and will continue to seek solutions to minimize time-consuming tasks and frustrations experienced by these faculty.
  4. We seek to quantify faculty usage of Canvas and its various tools and features in order to assess effectiveness of each to provide data to inform decisions on needed tools/features and future directions of technology.
  5. We recognize a distinct need to provide more direct support to students at the beginning of fall, winter and spring terms, and will determine how to leverage current resources to provide this support.
  6. Faculty are largely unaware of OSU’s Teaching and Learning Expectations, and the impact on course quality is currently unknown. It would be meaningful to develop a mechanism to measure impact as awareness and implementation of the Expectations increase in course design.