Evaluation

From Clinicaltools.com

Jump to: navigation, search

Contents

Why Is Evaluation Important?

It is important to evaluate our products so that we can be assured that they are of high quality and meet our objectives. To this end, we will collect and analyze a variety of data types, including knowledge that users gain from our courses, opinions about the ease of use of each website, attitudes about the health information presented, and skills gained from using our products.

Every project needs to be evaluated formally in some way, and every project needs to be able to report the evaluation procedures and results back to the funding agency (including successes, failures, areas for improvement, and how feedback was used to improve product quality). Continued funding often is based on both the process and outcomes of these evaluations, so we need to follow best practices and make sure that we accurately and fully report our results.

Types of Evaluation

  1. Formative Evaluation: a systematic assessment to determine how a product can be improved during the development process (this might include focus groups, usability tests, and needs analysis). These evaluations are usually part of Phase I but often are part of Phase II, as well.
  2. Summative Evaluation: a systematic assessment to determine the quality or utility of a product (this might include knowledge or attitude assessments at the end of a CME course, usability/accessibility analysis of one of our consumer sites, etc.). Summative evaluations usually are associated with Phase II.

Evaluation of Online Continuing Education

For online continuing-education courses, instruments assess the following:

  1. Knowledge. Using a post-test, users answer 3-10 questions based on the content of the learning experience.
  2. Self-efficacy. Following adult-learning theory and Bandura1, we assess self-efficacy related to the educational objectives for each individual learning experience.
  3. Attitude and Intended Behavior. The literature on physician behavior indicates that attitude heavily influences behavior, particularly in regards to screening- or intervention-related behaviors (reference needed). Thus, we feel it is valuable to assess whether the educational experience has influenced attitude and intended behavior. Self report is used for this measure (reference needed).
  4. Educational-objective assessment. Users are asked to rate whether they feel the online learning experience achieved the stated educational objective, using a five-point Likert scale.
  5. User satisfaction. Users are given a satisfaction survey after the experience. We feel that this simple measure is one of the greatest indicators of success from a product standpoint since satisfied users return to take more courses and tell their friends about the site.

Q&A

We collect a lot of data. What do we do with it, and how often?

Course data are reviewed at least annually to determine whether the course is meeting its established goals for usage, satisfaction, and change in self-efficacy, attitude, and intended behavior. Courses that do not meet goals may be revised or withdrawn, based on the judgment of the team. Factors include how far below goals a specific course is, whether there is variation in goals achievement among audiences (e.g., physicians vs. pharmacists), relevance/timeliness of material, and the course's role within a larger educational package, such as a curriculum or certificate program.

Who can see all of those data?

Data are compiled for each course and package of courses. Aggregate, anonymous data may be reported to the Accreditation Council for Continuing Medical Education (all data are available to the ACCME on request), the funding agency (if applicable), course authors, and Clinical Tools staff. The data may be presented at conferences, meetings, or scientific publications. Some data may be used for CTI outreach and recruitment materials, as well as for marketing.
Anonymous data may be made available in order to comply with Federal law regarding Federally funded projects.
Some non-anonymous data are available under specific conditions to the ACCME so that they may conduct audits and quality assurance, as well as verify physician participation.
By agreement, some non-anonymous data (such as participation and success) may be made available to third parties WITH the permission of the learner/user. For example, a physician may request that we confirm an inquiry from a state medical board or similar agency.

Why don't you conduct a case-control evaluation of your material?

Users resent taking a pre-test on the topic of the course, and they strongly resent taking a pre-test on non-related material (which would be used to determine whether "taking a test" influences the educational experience).

Steps in the Evaluation Process

  1. Create Initial Evaluation Plan. The PI or designee creates an evaluation plan for a project (i.e., evaluation design, logic, measures, instruments to be used, timeline, human subjects, etc.) during the grant-writing process. The evaluation plan will stem directly from the project's research questions and specific objectives.
    • Generate timeline early for each type of evaluation. This should be a month-by-month list of activities, along with who will have responsibility for each activity (e.g., IRB draft, recruiting, etc.).
    • Generate a Gantt chart to summarize visually the above information.
    • Generate a flow chart, showing how an individual would move though his/her part of the evaluation.
  2. Draft Materials for Evaluation. Based on the grant content and objectives, generate draft materials for the evaluation. Depending on the type of evaluation study you are conducting (e.g., focus group, usability study, summative course evaluation), the process, staff members, and materials will vary. Here are the general guidelines for both formative and summative evaluations:
    • Formative Evaluation (e.g., focus groups, needs analysis, early usability)
      • First draft of surveys
      • List of website elements to be tested in usability
      • List of issues/questions for needs analysis
      • List of issues/questions for focus groups
      • IRB proposal
      • Informed consent/confidentiality
    • Summative Evaluation (e.g., final CME course evaluations)
      • Pre- and post-test Items (see examples below)
        • Knowledge questions
        • Self-efficacy questions
        • Attitudinal questions
        • Behavioral questions
        • Clinical-skill questions
        • Satisfaction questions
      • First draft of surveys
      • List of website elements to be tested in summative usability
      • IRB proposal
      • Informed consent/confidentiality
      • Recruiting materials
      • Screening materials
      • Interview and observational protocols for usability
  3. Approve Materials and Procedures. After all first drafts are completed, they should be reviewed/copyedited by the project team and submitted to the IRB committee (allow at least two weeks for IRB evaluation).
  4. Implement Programming and Other Steps Prior to Evaluation.
  5. Review Entire Process and Materials. After programming is completed, the project team reviews the entire set of study procedures and materials to check for any problems. (Note: If there are any problems, they should be fixed; the project team then should run through the process again to be sure it is perfect.)
  6. Place Recruiting Ads. The project team posts ads to recruit participants, contacts participants to verify eligibility and set up evaluations, and ensures that the study population is representative of the target audience. Work on the assumption that recruiting enough subjects for a summative evaluation may take several months, especially if you are recruiting health professionals. Note: Ads cannot be posted until study is approved by the IRB.
  7. Collect Data.
  8. Track Progress. The project leader should track the progress of the evaluation and post significant dates on the company calendar.
  9. Analyze Data.
  10. Write Final Evaluation Report.

Note: Read your Notice of Grant Award to look for specific requirements about data collection, human-subjects policies, or the types of evaluation data you need; include those details in the IRB application. In addition, your evaluation will require some modifications to the courses as they normally exist (e.g., adding pre-tests, extending post-tests, adding consent forms, etc.). Contact the IRB if you have questions about whether a modification is needed. Do not change your instruments without keeping track of what was changed when. Make changes all at once, rather than in a stepwise fashion.

Sample Questions

Sample Knowledge Questions

The percentage of individuals with a mutation for a specific disorder who actually exhibit symptoms of the genetic disorder is called:
___ Penetrance
___ Variable expressivity
___ Sensitivity
___ Polymorphism
Correct Answer: Penetrance
Feedback: Penetrance is defined as "The proportion of individuals with a mutation causing a particular disorder who exhibit clinical symptoms of that disorder; most often refers to autosomal dominant conditions" (GeneTests.org).
Predictive genetic testing is designed to:
___ Identify individuals who may have a child with a recessive disorder
___ Identify a condition in a symptomatic person
___ Identify individuals who may develop a disorder in the future
___ Identify chromosome abnormalities in embryos
Correct Answer: Identify individuals who may develop a disorder in the future
Feedback: Predictive genetic testing refers to the situation where an individual is identified to be at increased risk to develop a genetic disorder in the future, they are not currently affected and they undergo genetic testing to determine their risk for developing symptoms. There are two major categories of predictive testing. The term presymptomatic testing is used when a person is being tested for a disorder that has nearly 100% penetrance and will, therefore, almost certainly develop symptoms in the future if found to have a mutation (e.g. Huntington disease). The term predispositional testing is used when an identified mutation increases the likelihood of a disorder but penetrance is reduced. This type of testing may be used to further refine one's risk to develop a disorder but is not diagnostic.

Sample Self-Efficacy Items (Usually Tied to Course Objectives)

  • I am confident in my ability to make basic tobacco use interventions with patients.
  • I am confident in my ability to discuss tobacco cessation issues with my patients (e.g., reasons for quitting, setting a quit date, and identifying challenges the patient might encounter).
  • I am confident in my understanding of the morbidity and mortality associated with using tobacco.
  • I am confident in my ability to make tobacco use interventions that are tailored to the specific needs of a patient.
  • I am confident in my ability to arrange appropriate follow-up appointments for patients who have decided to quit smoking.

Sample Attitudinal Items (Usually Tied to Evaluation Goals)

  • Motivating smokers to try to quit is part of my role as a physician (or other health care provider).
  • It is part of my role as a physician (or a designated health care provider or assistant in my office) to help a patient who is deciding to quit smoking to explore his or her barriers to quitting.
  • Physician interventions have little effect because smokers are just going to continue to smoke.
  • It is important to provide full informed consent regarding the benefits, risks, limitations and implications of testing to a patient prior to offering genetic testing.
  • I am confident that I know how to assess the benefits, risks, and limitations of genetic tests.

Sample Behavioral Questions

Current Behavior (Pre-Test)

  • I currently ask every patient 5 and older about tobacco use.
  • I currently arrange follow-up services for all of my patients who are trying to quit smoking.
  • I currently commend my patients who do not use tobacco.

Intended Behavior (Post-Test)

  • I intend to ask every age patient 5 and older about tobacco use.
  • I intend to arrange follow-up services for all of my patients who are trying to quit smoking.
  • I intend to commend my patients who do not use tobacco.
  • I intend to increase my scrutiny of the clinical utility and validity of genetic tests before ordering them for my patients.
  • I intend to provide informed consent for my patients prior to ordering a genetic test.
  • I intend to refer my patients who are candidates for genetic testing to a genetic counselor.

Sample Clinical-Skills Questions

A 20-year-old woman named Molly Black, who is seeing you for a chronic cough, thinks she might want to quit smoking. Which of the following is the best approach to motivating her to quit?
___ Use the same motivations as you would for a man
___ Tell her about her chances of getting lung cancer
___ Discuss motivations relevant to her cough
___ Encourage her to think of her unborn children
Your office assistant phoned Jessica Sweetbreath to remind her of the date she agreed to quit smoking. When should Jessica's next follow-up appointment be scheduled?
___ A week after quitting
___ A month after quitting
___ 6 months to a year after quitting
___ Whenever she feels the need
Your patient, Diane Gasping, has told you that she smokes and you have told her that you believe she should quit. Which of the following is the best next step in tobacco intervention?
___ Drop the subject and wait to see if she pursues it further
___ Show her a photo of someone who has had disfiguring surgery for a tobacco-related illness
___ Ask her directly if she is ready to quit smoking
___ Give her a brochure on nicotine replacement for her to consider

Sample Satisfaction Items

Generic

(Assessment)

  • _____ was easy to use.
  • The information in _____ is clear and makes sense.
  • The information in _____ is complete and thorough.
  • The information in _____ is well-organized.

(Impact)

  • There is a need for a _____ like this.
  • The information in the _____ is relevant to _____ [me, caregivers, my practice, etc.].
  • _____ was more useful than similar resources in finding information about _____.
  • After using the _____, I know something that I didn't know before.

(Intended Behavior)

  • I would like to use ____ again in the future.
  • I would recommend the _____ to other _____ (i.e. caregivers, patients, physicians, etc.).

CE/CME

  • The course content was clear.
  • The course content was free of bias.
  • The course was easy to use.
  • Overall this was a useful learning experience.
  • I would recommend that other health care workers take this online course.
  • The information in this course was relevant to my practice.
  • I plan to return to [SITE NAME] to take another online course.

References

  1. Bandura A. Social learning theory. Englewood Cliffs, NJ: Prentice Hall; 1977. 247 p. Amazon.com Listing

Resources

General Evaluation Resources

Statistical Resources

Textbooks, Tutorials, and Guides

Courses

Tools

  • RANDOM.ORG - "A true random number service that generates randomness via atmospheric noise"

HIPAA Resources

Retrieved from "http://wiki.clinicaltools.com/Evaluation"