Sample Usability Test Plan

From Clinicaltools.com

Contents 1 Purpose 2 Methods 2.1 Problem Statements 2.2 User Profile 2.3 Test Method 2.3.1 Participant Orientation 2.3.2 Performance Test 2.3.3 Participant Debriefing 2.4 Test Environment 2.5 Test Roles 2.6 Evaluation Methods 3 Results

Purpose

Clinical Tools, Inc's Continuing Education division has recently accredited several online Tobacco CE courses for ACCME, AAFP, and/or ACPE credit. The course content has been thoroughly evaluated for accuracy and completeness by CTI research teams, family practice physicians, and experts in the field of the course topic prior to accreditation. This usability test aims to evaluate the course delivery mechanism (the website). It will be evaluated for ease of use, ease of learning, and satisfaction of use by participants.

Methods

Problem Statements

1. Are users able to successfully navigate the website to find a specific course and complete that course?
2. Are users satisfied with the layout and organization of course content?

User Profile

Participants will be third and fourth year medical residents and physicians recruited through flyers in local hospitals and clinics. Recruitment ads will ask for research participants for a study assessing an online CME course on a tobacco cessation.

Test Method

The test will gather performance and preference information from approximately 10 participants. The test will provide data aimed at answering the problem statements. Participants will fill out post-test questionnaires and complete an interview to provide additional data to answer Problem Statement 2.

The usability test will consist of three main parts:

Participant Orientation

Participants will be greeted and read a brief orientation script. They will then be asked to fill out a pre-test form that documents demographic information and their previous Internet and CE experience.

Performance Test

The performance test consists of the tasks participants will be asked to complete while being observed by the test team. After the orientation is completed, participants will be seated in front of a computer displaying the TobaccoCME website. Participants will be told to take the CME course titled: Tobacco Cessation Overview and complete the course to obtain CME credit, but will not be given any further instructions. Following the completion of the course participants may be asked to complete 1 to 2 additional tasks, depending on the course being tested, that focus on innovative tools or resources offered on the website.

Participant Debriefing

After all tasks are complete, each participant will be debriefed by the test monitor. The debriefing will include:

• Filling out a post-test questionnaire to extract preference data concerning their attitudes about the site.
• The participant's overall impression and satisfaction with the website and course.
• The participant's comments to questions about events during the performance test.

After the debriefing session, participants will be thanked for their efforts and released. Participants will be paid for their time and efforts before they leave.

Test Environment

The tests will be conducted in the Clinical Tools' Inc., usability lab. We will create an environment similar to a work or home office, including a desk, computer, chair, and other typical office supplies. The PC will be sitting on the desk, will run Windows 98 OS, and have access to two Internet browsers – Internet Explorer and Mozilla.

Test Roles

The test monitor will perform all parts of the test, including pre- and post- test activities and the actual testing. The monitor will sit in the room with the participants while the test is conducted. The monitor will initiate the tasks and redirect the participants if necessary.

The test observer will observe the performance test and post-test interview. The observer will sit in the room, behind the participant during the test and record any timings, errors, observations, and talk aloud reports from the participants.

Evaluation Methods

The following evaluation measures will be collected and calculated:

1. Time to complete each task
2. Participant comments about the usability and design of the website
3. Data regarding participant satisfaction with the website experience
4. Error classification: The test monitor and observer's notes were reviewed to classify and identify each error. Error classes included:
   • Observed errors — The error was noted by the test monitor or observer but was not significant enough for the user to notice or to impair their use of the product.
   • Non-critical errors — The user noticed the error but was able to recover and successfully complete the task.
   • Critical errors — The error significantly impaired the user's ability to successfully complete the task. The user noticed the error and was not able to successfully complete the task.

Results

Preliminary results will be reported to the project team within 3 days of the last usability test. The full usability test report will be delivered to the project team within 10 days of the last usability test. Results will include quantitative results (time to complete tasks); qualitative results (observed errors, comments, interview answers); and participant satisfaction ratings (from the post-test questionnaire).