In an earlier blog we covered in-depth interviews, now I want to explore a structured form of research, known as Delphi. The purpose of the Delphi methods is to generate expert consensus in the formation of estimates and forecasts in operational planning and decision making.
It uses a controlled elicitation and feedback approach that goes through an iterative process and statistical analysis to reach a consensus that is not only statistically valid, but manages to incorporate outlying expert opinion in the development of that consensus. But I hadn’t heard of Delphi for many years, so when it came up recently in a discussion at conference, I wondered if it had fallen out of fashion and was making a come-back, or had previously been discredited in some way. So, here I've taken another look at Delphi to see if it had just been turn of the century fashion, or if it has anything to offer today’s researchers. And if it does offer something, explain more on how to use the Delphi method.
Investigating if Delphi was just a fashionable method
I'm going to tell you up front that is is not the latest thing in methods, but neither is it a fading fashion. It is a technique first developed in the mid-twentieth century, by the US RAND Corporation in the 1950s, (Dalkey & Helmer, 1962) to forecast the effects of warfare on technology; initially for expert informed operational estimates of numbers of bombs needed to destroy targets. However, Delphi has had many fields of application including health, public policy, social equity, business and law, and education. To assess if it is a fading fashion, I needed a way to gauge the use of Delphi over this time. All research outputs have increased over time, so I compared search results for Delphi and a comparator qualitative method, Research focus group method.
I anticipated that Focus Groups would be a more common choice in research and therefore appear more often in Google Scholar search results, and that both would increase over time. What I was interested in was any changes in the gradient of the curves for my two search terms “Research delphi method”, and “Research focus group method”.
Chart 1 Comparison of article numbers in Google Scholar for Delphi method and Focus Group method between 1962 and 2023
This crude count of Google Scholar search results (Chart 1) indicates that the count for both methods has increased, but focus group method increased around the turn of this century, much more than it did for Delphi.
Top results in Google Scholar for Research Delphi Method in 2023 come from diverse areas including clinical, psychology/psychiatry, dentistry, maritime accident investigation, blast analysis, supply-chain management, medical and health education, sustainability, applied linguistics, and supply chain logistics. Bear in mind however that this is not a review of the literature, just counts of results from a Google algorithm.
It seems from these illustrative results that Delphi has always been less favoured in qualitative circles than focus group method and its growth in use has not increased as much as focus groups. This is unsurprising to me as it is a very particular method used for particular purposes, and as such, lacks the greater flexibility of a more general approach like focus groups. Delphi interviews will always be structured, but focus group interviews have evolved to be structured and semi-structured, almost to the point of having no structure in some creative and participatory approaches. But am I being a bit biased here?
What does Delphi offer the researcher of today?
The reason I hadn’t come across Delphi for so long was not necessarily because of a decline in use but because as a method it didn’t suit the purpose of the types of research I was doing, rather than any deficit in the Delphi method. (I was using methods that were much more suited to phenomenological and humanistic approaches, such as unstructured and semi-structured interview).
The purpose of Delphi methods is directed elicitation of convergent expert opinion to assist in the formation of estimates and forecasts for use in operational planning and decision making, i.e. expertise informed consensus. The earliest explanations of its purpose remain relevant today (even though the gendered language isn’t):
“Designed to bring out the respondent’s reasoning that went into his reply to the primary question, the factors he considers relevant to the problem, his own estimation of these factors, and information as to the kind of data that he feels will enable him to arrive at a better appraisal of these factors, and thereby at a more confident answer to the primary question.” (Dalkey & Helmer, 1962: 1-2).
Newer studies have a wider remit i.e. the goal is broadly understanding consensus within a topic. RAND Corporation researchers continue to adapt and develop the technique in their research. If you are a student of security, international politics or related public policy, you will almost certainly have heard of them, but they also research in areas in which the Delphi method continues to be used (e.g. Education, Health and Medicine, Environment, Workforce and Infrastructure, and Law) with such purposes as supplementing climate modelling, hypothesis generation, stakeholder perspectives, priority-setting, and facilitating engagement in citizen science. According to Hirschhorn (2019), Delphi can produce a breadth of views that makes it almost unparalleled as a building block for continued and more in-depth analysis than workshops, interviews or case studies alone.
How to do the Delphi method
Humphrey-Murto (2020) et al give a very readable model and overview of the method in clinical research, as does Chuenjitwongsa (2017), (see Resources). Even beyond clinical studies, four core elements have evolved in the Delphi method: the development of an aggregated response through an iterative process of controlled feedback to anonymous panel of experts. The gathering of expert opinions is usually via survey or questionnaires means, with simple parametric statistical analysis of findings, followed by controlled feedback of results to experts by a skilled facilitator. Surveys are usually numerical or scalar, but in keeping with Dalkey & Helmer’s original experimental survey (1962), there is also an emphasis on identifying or clarifying what factual or contextual information has led the expert to the numerical value given, and/or what new information/context would change the value given. The generalised model below gives greater detail of the steps in the process.
Quirkos generalised model of Delphi method process (Gibbons 2023)
As Hirschhorn (2019) points out, “Although as initially conceived the Delphi was not meant as an open and inclusive participatory process – and instead as a methodology to achieve consensus among a small and selected group of experts – over time new variants of the method appeared and opened up for inputs from more actors as well…. The Delphi technique is particularly well suited to solve complex and multi-layered problems that require the attention of multiple stakeholder groups.”
Not only have more creative and inclusive ways to ensure inputs been developed, but adaptations that would seem to challenge the generally described 'core elements' of anonymity and the use of statistics to aggregate responses. These variants adapt the method in different ways, such as techniques to select participants, types of questions employed, quasi-anonymity of the experts, qualitative tools used for the analysis of responses, and differences in the type of outcome sought e.g. perspectives not consensus. At the other end of a perceived ontological spectrum approaches that prioritise statistical validity remain commonplace. Robust statistical forms such Fuzzy Delphi and Large-Scale Group Delphi Method (LSGDM) are also emerging.
Meeting the epistemic challenge in the requirement for expert knowledge.
Delphi certainly seems to be a structured approach but where Delphi sits in terms of ontology and epistemology is debateable. Dalkey & Helmer’s 1951 experiments had only seven experts and they acknowledge the statistical limitations and epistemic tensions in their 1962 report. By the very nature of expertise and niche nature of some problem topics, large numbers of well-informed individuals are not available for statistical surveys. Reaching experts, maintaining their input, consolidating responses in a clear, concise fashion but avoiding generalisations which would defeat the purpose of eliciting expert opinion, requires having a skilled co-ordinator to facilitate articulation of opinions and dialogue between experts. Outlier opinion is not rejected as in other statistical models, but re-integrated and represented alongside consensus views to enable re-examination and potential reformation of the consensus view. The anonymous nature of the presentation of questions and feedback of data avoids the psychological drawbacks of face-to-face discussion such as halo effects, persuasive and charismatic effects, agreeable and disagreeable personalities, and tendency towards group-think.
Delphi is qualitative in terms of it being an iterative process, with feedback to informants, and some of the data being narrative. However much of the data is structured to be numerical and analysis is statistical, albeit a simple parametric analysis. But ontologically, the context and the uncertainty of the future (hence why the original project was called Delphi after the Greek oracle) calls for a level of intuitive probability estimates (i.e., an educated assessment of the likelihood of X happening) that can sit uncomfortably in a positivistic paradigm. Delphi method attempts to meet the requirement that operational decision making and informed planning needs to take account of less-likely but high-impact events. From a qualitative perspective, statistical survey results are critiqued as they focus minds on commonly held views, with little informed consideration of minority views. Dalkey & Helmer’s (1962) invitation to future researchers to build more robust methods continues to be answered, but with differing responses.
Where data is numerical or scaler in nature and definitions of consensus are defined with reference to descriptive and inferential statistics (such as mean and median values, and standard deviations), one wonders if Delphi could ever be used qualitatively. However, not all Delphi studies will gather numerical data (mini-Delphi). Methods may also be quasi-anonymous where face-to-face gatherings are used to facilitate the exploration of difference in expert or participant opinion. Delphi method is just that; a method; a tool of process to facilitate analysis. Given the potential to mix numerical data with text data, and to mix the contextualised knowledge from the subjective positions of different specific informants with statistical analysis, I think Delphi qualifies as a mixed-method approach.
Conclusion
Perhaps we’ll explore in more depth what makes a study mixed-methods in a future blog. But if you are looking for a tightly defined process that can be worked through to identify consensus, using statistically valid methods, then the classic Delphi method could be for you. If you need something participatory, iterative and very process oriented in data gathering then I invite you to consider quasi-Delphi methods. It may not be as popular as its qualitative cousin, Focus Groups, but with growing adaptability of purpose and context. I think it’s here to stay.
This article first appeared on the blog for Quirkos 22 Aug 2023. https://www.quirkos.com/blog/post/structured-interviewing-is-delphi-a-fading-fashion-or-still-a-relevant-method-in-qualitative-research/. It is reproduced here with the kind permission of Daniel Turner, with minor adaptations. Director of Quirkos. I am not sponsored by Quirkos to reproduce this blog; any recommendations are truthfully my own.
Resources
Examples of RAND Delphi projects
How to use the Delphi Method
Humphrey-Murto, Susan MD; associate professor; Wood, Timothy J. PhD; professor; Gonsalves, Carol MD; assistant professor; Mascioli, Kelly MD; lecturer; Varpio, Lara PhD; associate professor.
Thumbnail image only: to view detail visit original https://www.tandfonline.com/doi/full/10.1080/13645579.2018.1543841
Academic Medicine 95(1):168, January 2020. https://journals.lww.com/academicmedicine/Fulltext/2020/01000/The_Delphi_Method.41.aspx or
How to conduct a delphi study. Supachai Chuenjitwongsa. (2017) How to: Conduct a Delphi Study. How to: Series. Wales Deanery/Dioniaeth Cymru. Cardiff University.
Thumbnail image only: to view detail visit original https://www.cardiff.ac.uk/__data/assets/pdf_file/0010/1164961/how_to_conduct_a_delphistudy.pdf
Other Delphi papers
Maite Barrios, Georgina Guilera, Laura Nuño, Juana Gómez-Benito (2021) Consensus in the delphi method: What makes a decision change?, Technological Forecasting and Social Change, Vol. 163. https://doi.org/10.1016/j.techfore.2020.120484.
Fontana, A., & Frey, J. H. (2000). The interview: From structured questions to negotiated text. In N. K. Denzin, & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 645-672). Thousand Oaks, CA: Sage.
Fontana, A., & Frey, J. H. (2000). The interview: From structured questions to negotiated text. In N. K. Denzin, & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 645-672). Thousand Oaks, CA: Sage.Fabio Hirschhorn (2019) Reflections on the application of the Delphi method: lessons from a case in public transport research. International Journal of Social Research Methodology, 22:3, 309-322, DOI: 10.1080/13645579.2018.1543841 To link to this article: https://doi.org/10.1080/13645579.2018.1543841
Susanne Iqbal and Laura Pipon-Young. (2009) The Delphi method with a step-by-step guide. The Psychologist. Methods. British Psychological Society. Vol 22. no. 7.
https://www.fws.gov/sites/default/files/2022-02/Iqbal%20%26%20Pipon-Young_2009_The%20Dephi%20method.pdf[and abridged July 2009 online https://www.bps.org.uk/psychologist/delphi-method]
Natalie Jago (No Date) Mental Health Beyond the School Gate: Young People’s Perspectives of Mental Health Support Online, and in Home, School and Community Contexts. BPS Online. https://www.bps.org.uk/mental-health-beyond-school-gate-young-peoples-perspectives-mental-health-support-online-and-home
In Farsi
Rahmani A, Vaziri Nejad R, Ahmadi Nia H, Rezaian M. (2020) Methodological Principles and Applications of the Delphi Method: A Narrative Review. J Rafsanjan Univ Med Sci 2020; 19 (5): 515-38. [Farsi]
Comments