At the 2008 International Association of Medical Science Educators (IAMSE) conference, a presentation was given by Stephen Davis using the Delphi Method of focus groups. We would like to use this methodology at future medical educator conferences to solidfy our standardized patient methodology and affirm the direction of our medical student projects.
What is the Delphi Method?
According to Wikipedia, the Delphi Method is an interative process designed to come to a consensus after iterative rounds of discussion. Taking a panel of experts, the moderator asks a series of questions. After each round of questions, the answers are discussed, and the questions revised and discussed again. This iterative process continues until the stop criterion are met. Ultimately, a consensus on the questions/answers is reached after the last iterative round.
"The Delphi method is based on the assumption that group judgments are more valid than individual judgments." Wiki: History The main premise of using the Delphi Method to answer questions is that participants can learn from the responses of others, and together, through iterative discussions, arrive to a decision on the question at hand (consensus).
The Delphi Method consists of the following (from Wiki):
- Structured information flow
- Regular feedback
- Participant anonymity
A report by Swedish researchers gives a very good summary of using the Delphi Method in research.
Example Uses of the Delphi Method
Delphi Method in Research
- Ndour B, Force JE, McLaughlin WJ. Using the Delphi Method for determining criteria for agroforestry research planing in developing countries. Agroforestry Systems. 1992; 19(@):119-129. - Abstract
- Howze PC, Dalyrmple C. Consensus without all the meetings: using the Delphi method to determine course content for library instruction. Reference Services Review 2004; 32(2):174-184 - Abstract
- Katcher ML, Meister AN, Sorkness CA, Staresinic AG, Pierce SE, Goodman BM, Peterson NM, Hatfield PM, Schirmer JA. Use of the modified Delphi technqiue to identify and rate home injury hazard risks and prevention methods for young children. Injury Prevention. 2006; 12:189-194 - Abstract. Used 3 round delphi survey w/experts, through e-mail. Got demographic data from participants, knew who they were, areas of expertise. Blind survey in that participants didn't know each other.
Methods: Survey 1 - Requested list of hazards for children 1-5 in areas of the home. Survey 2 - Rated each hazard using a priority scale. allowed a score of 0 if they didn't think item belonged on list - mean scores calculated and list reformalized by order of mean score Survey 3 - Rate each hazard based on likert scale of importance to injury prevention. Determined final rank order by % of those in top 10
- Weerakoon P and Knight P. The use of a modified Delpi Method to determine global trends in sexology education. Medical Education. 2008; 42(3): 324.
Methods - modified Policy delphi - "identify common goals for graduate secology education" - put out research question to experts, get responses back. Discuss through several iterations to get options, then discuss F-T-F 4 questions - logical progression of curriculum development (aims, content, modes of delivery, modules) one question on open source learning attitudes, another on collaborations.
Use of Delphi Method in Conference Presentations
A presentation [Exploring Personal Computer Use in the Lecture Hall and Small Group Settings, Stephen Davis, Ohio University College of Osteopathic Medicine] was given using the Delphi Method of focus groups. His discussion centered around identifying strategies and tactics of dealing with individual laptop use in the classroom. Mr. Davis had hoped to use a Delphi-like method to "flush out and prioritize actions for educations to deal with this new reality". He had these expectations:
- Produce a list of assets and liabilities of student laptop use during class
- Produce rules of engagement to exploit assets and minimize liabilities
- Facilitate discussion of possible research questions to scientifically study the challenges and opportunities of student laptop use during class
The above presentation successfully used Dephi method (similar to a anonymous focus group) to answer the question "what are you going to do as a professor/lecturer when your students use laptops." The purpose of this group was basically to define the questions. The group came up with
- how do you incorporate the technology (e.g. do a search during the lecture so show medical informatics) and
- how do you take control and tell folks to behave appropriately.
There wasn't really time to answer the questions. For many this was not satisfactory since they wanted to go beyond defining the problem, but the method is iterative and thus this is not a reasonable expectation. The goal is not to produce the equivalent of a definitive lecture on the topic (immediately).
The presentation could have worked better if it could have started further into the process and if everyone already understood the problem. Much time was spent getting everyone up to speed with what the problem/question was. In this case the problem was that medical school administration was forcing technology on the professors since every student must have a laptop. The laptops allow the students to ignore the existing presentation style and demand that the lectures do something differently.
Given the topic it was strange that only one other person in the audience actually had a laptop that they used during the process. So success potentially also requires a willingness/interest to see the problem from multiple perspectives (in this case from students as well as lecturers), or to include a more diverse group of participants.
Using the Delphi Method with our Medical Education Projects
- Hypothesis - We propose to use virtual standardized patients as an intergral part of the medical student learning experience. Through our curriculum, medical student will gain valuable experience interacting with standardized patients. This emphasis on skills training, in addition to the knowledge gained through the curriculum, will empower students in their rotations and in residency.
- Needs Analyses - We have conducted surveys with medical students, residents, and practicing physicians in an attempt to define aspects of the virtual standardized patient (mode, length, etc). We need to hear from the medical education community their thoughts on including virtual standardized patients within the learning experience and how this novel method of teaching will integrate into current educational platforms.
- Methodology - We propose to use conference presentations as a focus group to determine the boundaries of our virtual standardized patient program. Using the Delphi Method, we will ask iterative sets of questions of the participants and come to a consensus about the long-term viability and utility of our proposed SP program.
Using a Modified Delphi Method to Obtain Consultant Feedback on CME Projects
We modified the Delphi method by making the process asynchronous and used it to get feedback from expert consultants on CME project components, such as frameworks and curricula. As with the Delphi method, we seek feedback from consultants throughout development of project components. Each consultant individually provides feedback directly to our research team on initial draft descriptions of a component that is sent to them. Feedback is provided in response to open-ended questions sent via email, an online survey, or an interview. After each iteration of feedback, the project component being reviewed (e.g., framework, curriculum outline, cases) is revised and returned to the consultants and they are provided an opportunity to respond again, and so forth, until a consensus is achieved for that round. An advantage of this modified method is that the best experts, who are often very busy, can collaborate asynchronously and develop a consensus. Another advantage is that feedback is provided anonymously and may be more likely to represent actual views on a topic.
Current Data to Include
Alcohol Med Student Data The student responses to our needs analysis confirm that students place value on case-based learning, as a means to reinforce didactic learning and practice patient skills. Students deem it worthy to have constant access to pertinent information in order to strengthen the learning experience and further information comprehension. Three cases was sufficient for an hour-long CME course.
In terms of using virtual standardized patients, the students thought that on-line chat was a good venue to use. The students seemed willing to use the standardized patients as part of the learning experience.
Students indicated that the topics in our curriculum are indeed ones that they would like to receive more information on.