Usability Testing

From Clinicaltools.com

Jump to: navigation, search

Contents

Live Usability Testing

Planning

  1. The project team determines the quantitative and qualitative data it wants to retrieve from usability testing.
    • Quantitative data include:
      • Time to complete tasks
      • Ability to find key information
      • Ability to complete key tasks
      • Ability to navigate website
    • Qualitative data include:
      • User perception of website
      • User satisfaction with website
      • User suggestions to improve website
  2. The moderator and project team work together to set measurable goals for the usability test. An example of one such goal would be: "Subjects should be able to find the Resources section within 10 seconds of starting to look for it."
  3. The project team determines the dates/weeks of testing (based on when any deliverable is due).
  4. The moderator prepares the printed materials, which include:
    • Consent forms for participants 21+
    • Assent forms for participants under 21 (and consent forms for their parents)
    • Demographic/habit questionnaire
    • Moderator's guide
    • Scripts
    • Scenario and task lists
    • Interview questions
    • Post-test questionnaire

Recruitment

  1. The project team develops a screening instrument.
  2. Team members post ads, contact participants to verify eligibility and set up appointments, and ensure that the study population is representative of the target audience.
  3. The following are possible advertising venues:
  4. The following is Clinical Tools' compensation schedule for study participants:
    • $40.00/test - General public
    • $50.00/test - Member of a specific demographic group (e.g., medical students, seniors, nine-year-olds)
    • $75.00/test - Master's-level professionals
    • $150.00/test - Doctoral-level professionals

Testing

  1. Conducting the test takes approximately 1.5 - 2.5 hours. This includes set-up, greeting and orientation, testing, and conclusion.
  2. The observer reviews the website and printed materials before testing and meets with the moderator to discuss any questions/concerns.
  3. The moderator and observer ensure that:
    • There are plenty of chairs and tables in the testing environment;
    • The chairs are short enough so that the observer can see the monitor;
    • The computer uses a traditional/flat keyboard with a wired mouse;
    • The computer is properly connected to the Internet;
    • All of the printed materials (see above), checks, and tape recorders (if used) are ready; and
    • The ambient temperature is comfortable.
  4. Dress appropriately, in accordance with the Clinical Tools Dress Policy. Please do not wear jeans, tank tops, short skirts, or visible piercings (besides ears).
  5. The study is conducted with both the interviewer and the observer present for all sessions, if possible. The Project Director is present for one session, if possible, as an observer.
  6. All moderators and observers take notes during each session.

Observer's Role

During Testing

  1. The observer is in charge of taking notes during a usability test.
  2. The observer should not speak, make facial expressions, or suggestive noises during usability testing.
  3. At the end of testing, the moderator will ask if the observer has any questions.
  4. If the observer does have questions, he/she will form them in a non-leading manner.
  5. The observer will give the subject his/her check. He/she will give the check portion only and will keep the bottom two copies (with the participant's name) for Clinical Tools' records.

Note-Taking Process

  1. Write like crazy. Write down all navigational paths and ALL comments.
  2. Comments can be paraphrased, but try to write as many verbatim as possible.
  3. Underline the links that users click.
  4. Use arrows (>) to denote sequential links clicked without comment.
  5. If comments are made, underline the link and then write the comments that users make next to that link.
  6. Notes should be submitted to the moderator within 24 hours of the test. This ensures accuracy (as the information is still fresh) and ensures that the notes can be discussed, if needed. Note: Typing up notes takes, on average, twice as long as conducting the test.

What to Watch

  1. Links clicked
  2. Facial expressions
  3. Mouse movements
  4. Use of "back" and "forward" buttons or other navigation
  5. Whether users realize which links are external
  6. Amount of time taken to complete a task
  7. Any pauses or tone changes

Close-out

When the study is complete and the data are entered, all notes are analyzed and combined into one set by the moderator. This usually takes about an hour, depending on the degree of variation between the notes. The notes include:

  • Overall user performance and preference data
  • Tasks that users performed well
  • Parts of the site users liked
  • User performance problems, the cause of each problem, and ideas for possible solutions
  • How user preference could be improved

A list of recommended changes to the site is then created.

Usability Report

The moderator then writes a draft of the usability report and submits it to the project team for questions/comments/revisions. The time to write the report varies drastically by project, depending on the amount of errors found, the project team, the readership of the report, etc. A typical report can take anywhere from one day to two weeks to write, edit, and finalize. Usability reports include the following sections:

  1. Executive Summary
    • This is a one-page summary of the highlights (think of it as the business world's version of the academic abstract). It duplicates some information in the Introduction (see below), but that is precisely the point: the Executive Summary should be able to stand on its own. Usability testing was performed on July 14 and 15 in order to assess [Site Name]....Nine users were....Significant findings include....successes....improvements....
  2. Introduction
    • What is this report about? What are we going to tell them?
    • What was our main goal?
    • What was the main thing we did?
  3. Method (Full Description)
    • Just what exactly did we do with the subjects?
    • Mention the dual participation of the moderator and observer, the number of tasks performed, and what the tasks were.
    • Include the demographic information.
    • Put the interview guide in an appendix.
  4. Results
    • Briefly summarize the raw data. For each question/task, provide a brief, one-sentence description of the task and a summary of the data. This is not the place to analyze; it is fine to say something like "this was a trouble area due to the lack of clear direction," but leave the proposed solutions for the Discussion.
  5. Discussion
    • Divide this section into two parts: Where [Site Name] Succeeds and Where [Site Name] Needs Improvement ("Identified Problem Areas"). Use a Problem/Solution set-up for each issue and include the "Status of Solution" (e.g., "Done," "In Progress," "Expected Completion: 2008-02-01," etc.).
  6. Conclusion
    • This can be one to two paragraphs, repeating the main themes brought up in the report and recapping the highlights from the Executive Summary.

Online Usability Testing

Background

In 2005, we conducted in-house focus groups for the SeniorDepression.net project, for which we recruited older adults (55+) who had been diagnosed with depression, as well as caregivers of this population. However, these populations proved extremely difficult to locate in the Research Triangle area and even more difficult to schedule and bring into the office. These difficulties were reflected in the mediocre results and disinterested attitudes we encountered. We then decided to attempt an online focus group, which proved much more fruitful.

About Online Usability

Online usability is the practice of conducting usability or other evaluative testing over the Internet. There are a variety of ways to do this, including the use of software or Web-conferencing programs that allow parties to share control of a single browser window. Here at Clinical Tools, we use a telephone-Web method that allows participants to evaluate a website in the comfort and privacy of their home/office — without special software or equipment.

Positive Aspects of Online Usability

  • Participants can come from all over the United States.
  • Participants can evaluate websites from their homes or offices and do not have to travel during business hours.
  • With a wider participant pool, we can select users who more accurately meet our demographic needs.
  • CTI's session data can provide a completely accurate navigational path, including the time spent on each page.

Negative Aspects of Online Usability

  • Participants must have Internet access independent of their telephone (i.e., no dial-up) in order to use both simultaneously. This can limit your sample. However, high-speed Internet is much more common these days.
  • You cannot see participants' facial expressions.
  • Much of the evaluation depends on what users tell you, not what you observe.

Although online usability is not fool-proof, it can be useful for getting a general idea of a website's usability. By asking detailed questions—including asking users to describe what page they are on—one can determine areas of difficulty.

Resources

Websites

Books

  • Flagg BN. Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum Associates; 1990.
  • Jones MK. Human-computer interaction: a design guide. Englewood Cliffs, NJ: Educational Technology Publications; 1989. 150p.
  • Keeler JM. Motivational design. In: Unwin D, McAleese R, eds. The Encyclopedia of Educational Media, Communications and Technology. 2nd ed. London: Kogan; 1988:404-9.
Retrieved from "http://wiki.clinicaltools.com/Usability_Testing"