Section 3: Online Instructional Materials and Course Organization

Checking for Usability by Beta Testing

The easiest way to check the usability of your course is to do a bit of usability testing. Usability testing at this level can be as simple as having a small handful of people who are not familiar with your course perform 3-4 common tasks while you observe them and the difficulties they experience. Seeing your course and materials from the perspective of someone unfamiliar with your content and organizational structure can help you to clarify important instructions, catch grammatical mistakes, and generally provides a useful second set of eyes to help you see things you hadn’t noticed before.

Usability testing can help you answer questions like these

    • Do your page titles accurately describe page content?
    • Are your content pages, assignment instructions, etc. written in an active, 2nd-person voice?
    • Are your content pages, assignment instructions, etc. free of grammatical and spelling errors?
    • Do your pages, syllabus, assignments, etc. use built-in heading styles to help readers follow your organizational structure?
    • Are your instructions clear to people with a student-level understanding of the concepts?
    • Do any provided rubrics, models, or examples actually help students understand what they are supposed to do?
    • Are all links in your content and instructions working and do they use descriptive text? (ex. IUPUI Homepage instead of https://www.iupui.edu/)

While all of these questions are important, you can check many of these yourself, such as whether or not you are using headings or 2nd person active voice. Things that you may not notice that are very helpful to have someone else check, include:

  • Navigation – Sitting someone down on your course home page and seeing if they can find what they need to find to do what they need to do can be an eye-opening experience.
  • Instructions – Silently sitting with someone and having them talk through what they think they should be doing based on your instructions is an easy way to find gaps. While you know how to do something and don’t think it needs explaining, students may or may not know how. This is especially true if you have a mix of majors and non-majors, or continuing and new students. In addition, while they are not technically proofreading – this kind of walkthrough will also turn up grammar and word usage issues as they try to make sense of what you are asking them to do.
  • Links – Can they reach the link target? Sometimes links are added that work for you because you are logged in to a particular website or database, or because you have a higher level of authorization than your students. Having someone else who is only logged in as a student test your links can help you find any that may cause problems.

While someone is walking through your navigation and instructions, and talking through their thought process, you will also often get additional user experience feedback. Encouraging them to share what they are feeling as well as thinking can give you information on things that cause frustration and annoyance or engagement and interest. 

Beta Testing

Beta testing is usability testing done at an early stage in something’s development. You may think of as only being for apps, video games, or other computer programs but beta testing is also a useful process for online courses.

When beta testing a course or a section of a course, the tester enters the course as if they were a student in the course and tries to do what a student is supposed to do. Whether or not they can figure out what they are supposed to do is the first hurdle. While they obviously won’t be reading a textbook, they will read the syllabus, take a syllabus quiz if you have one, introduce themselves, watch your videos, try to access any external resources, as well as post in discussions and/or submit mock assignments when feasible.

Testers

It is a common process for MOOCs such as those hosted by Coursera to have 25-30 people beta test a course before it goes live. Faculty are not likely to have access to that many people to beta test a course but it is good to have at least 2-3 people test it if possible. Beta testing is most useful when the people testing are members of your target audience. Is your course for undergraduate or graduate students? Is it for majors only or can anyone take it as an elective? Are your students commonly traditional-age students or do they tend to be working adults? Students who have previously taken a different in-person course with you are likely to give you the best information. You don’t want students who have taken this particular course before as they will be familiar with the content and may not review it as thoroughly as someone who has not seen it before.

Testing Options

The key to useful feedback is to have your testers actually use the course while you observe and have them provide feedback. Don’t ask your testers whether they “like” something or not. People will often say they “like” things while also being confused by them. As Margaret Mead noted, “What people say, what people do, and what they say they do are entirely different things.” In beta testing, we want to know what they do, why they think they do it, and how they feel about it.

“What people say, what people do, and what they say they do are entirely different things.”

You can conduct beta testing either synchronously or asynchronously. There are two common ways to do synchronous beta testing, in-person and virtual. In in-person testing, beta testers come in to test in person individually or in a group while virtual testing is conducted in a video conference meeting or another form of live video observation.

The benefit to synchronous testing – either in-person or virtually – is that you can see when the tester is confused or frustrated and ask them to describe what is going on in the moment, what they are thinking and feeling. A potential drawback is that your presence in the same room or on the other end of a video call may distract the tester. They may attempt to ask questions of you to clarify things before trying them for themselves. Conversations can develop explaining parts of the course before testing occurs when you prompt them to explain what they are seeing, thinking, or feeling. It is important to make sure you are only observing, not guiding or biasing their perceptions of the course. 

The benefits of asynchronous testing include the convenience for the tester and the diversity of systems that you can test with (i.e. Mac, iOS, Windows, Android, different size displays, using the Canvas Student app, etc.). Testers can “take the course” as an asynchronous student would at their own time and place. Since they are testing using their own computer or mobile device, it allows testers to surface technical issues that may be specific to a certain operating system, browser, or screen size. The richest data come from asynchronous testers recording their screen and thinking aloud about what they are doing and feeling as they would in a synchronous testing situation. If that isn’t possible, for example, if a tester is using a mobile device without a screen capture app, it is common to ask the tester to log what they are doing as they are doing it, any issues with or questions about the course, and how they feel about what they encounter.

Whether you are doing synchronous or asynchronous testing, compare what your testers actually do to what you thought they would do to check your own mental models of how students would “naturally” experience your course. One way to surface some of these experiences is to have testers restate instructions for assignments in their own words to see how close their interpretation is to your intent. It is also good to have a standard set of questions that you ask your testers at the conclusion of testing to get focused feedback on critical points where you may already have concerns. 

What Next?

Once you have information from your beta testers, you revise. Mechanics like spelling, grammar, and broken links can be fixed quickly and easily. Once those are taken care of you can turn your attention to the remaining feedback and look for commonalities across the testers. If more than one person felt confused or frustrated at the same point or had questions about the same assignment, you likely need to adjust the placement or formatting of something or revise the text or video. The final step is looking closely at the challenges only one person brought up. Sometimes a thing brought up by one person were noticed by others but they may have felt it not worth bringing up. For example, visual design is an important part of organizing text on a page. While others may have disliked a page full of text with no headings, it may that only one person could articulate what about the page was frustrating.

 

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Online Course Development Basics by Trustees of Indiana University is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.