Examples of Delphi structures

From FORwiki

Jump to: navigation, search

There are a number of "classic" structures that have been used very successfully many times in the past forty years and have been the basis of a number of proprietary organizational studies. They can each be used on a wide range of similar problems. Some of them have been utilized in online exercises using bulletin boards and auxiliary software such as survey packages. A few have been fully implemented in software. These are the original Policy Delphi structure[1] and most recently the Problem Solving Delphi[2].


Delphi structures

An online Delphi is extremely dynamic and what used to take months using paper and pencil processes can now be done in a few days or a few weeks. Some particular urgent real time problems such as those in the area of Emergency Management can be dealt with in less than an hour by small groups of 10 to 20 dispersed participants. In an online continuous Delphi process, any participant at any time may:

  • Focus on the particular topic they want to deal with
  • Be informed of changes since he or she was last using the system
  • Be able to create new material, new comments, new options to vote on, etc.
  • Be able to vote or change their vote
  • Defer voting until there is more information about an item to make a judgment
  • Allow lists of items to which additions can be made or current items edited
  • Arrange such lists dynamically according the attached voting scale results
  • View vote results to show differences by voters with different backgrounds
  • Provide dynamic collaborative tagging to create new classifications of items and lists
  • Show the status of voting with respect to number voting and vote changes for each item as well as the vote distribution.

General types of Delphi processes

The following are the general types of Delphi processes that apply to a large number of applications:

  • Trend Delphi: produces a forecast of a trend along with the mental model of the group making the extrapolation of the trend curve into the future.
  • Problem Solving Delphi: Collects solutions to the problem which are rescaled to a group interval scale based upon individuals ranking or paired comparisons. Use voting to focus discussion on items that need it.
  • Policy Delphi: seeks policy resolutions and the strongest pro and con evidence or arguments to support each policy resolution.
  • Cross Impact Modeling: Collaborative building of a model of the future possible outcomes of a set of unique events.

Each of the above has specific characteristics that are summarized in the following tables:

Trend Delphi

System Functions Participants Responses System Actions on Responses
Present a historical trend to be extrapolated by the participants Draw a future curve or redraw a new one when a change has occurred in viewpoint. Present summary of 50% median and 0%, 25%, 75% and 100% boundaries
Request assumptions and uncertainties used to make above estimate; Turn all these into potential assumptions Vote on validity scale for each potential assumption. Scale is from completely true to completely false. Reorder assumptions from true to false. Focus on middle range (maybe) and ask which can be influenced or measured for occurrence
Assume these can reduce the future uncertainty in the curve; Ask for a redrawing of curve extrapolation based upon assumption list for each trend curve in the study Supply suggestions on how to influence or measure the "maybe" assumptions causing significant uncertainty in the projected curve. Summarize important findings at any time: trend, true and false assumptions, assumptions that cause uncertainty, and their potential actions, and measurements

Problem Solving Delphi

System Functions Participants Responses System Actions on Responses
State the problem and request solution options Provide options to solve the problem Present options in order of occurrence
Request paired comparisons to measure individual preferences for options Make comparisons for option pairs that a participant feels confident about judging at any time. Use Thurstone's law of comparative judgment (using incomplete information) to derive a single group interval scale. Calculate uncertainty due to those who have not yet voted with same type of scale.
Show interval scale: this indicates disagreements when two or more items are close together. This also shows clustering. Ask for comments about items where people disagree from vote. Make comments about items you want to see others change their votes about. Present discussions about items for review. As more people vote or change votes, scales will reflect decreasing uncertainty and often more separation between options.

Policy Delphi

System Functions Participants Responses System Actions on Responses
State a policy issue to be examined. Ask for specific policy solutions Add resolution options or specific policies Request vote for Desirability and Feasibility scales of each solution
Plot two dimensional distribution of policy resolutions; Exploring desirable but infeasible solutions often important Request comments especially about those showing disagreement Request comments about policy resolutions. Indicate if comment is pro, con, or neutral.
Request vote on comments for importance and validity (It might be considered important because others believe it to be true) A person may think a comment is important because others think it is valid. Provide updated two dimensional plots and summarize discussions

Cross Impact Modeling

System Functions Participants Responses System Actions on Responses
Use problem solving Delphi to produce a set of future unique events focused on a given situation Evaluate those events for their relative importance to the future objective guiding the choice of events Place the final most important events into a cross impact model
Ask each individual to answer: what are the probabilities of each event occurring in some future time frame? Tell them for each event that they should assume it will or will not occur and ask them to express any changes in the probabilities of the other events due to that certain knowledge about the future. Show them the expected outcome of their model, which will have differences from their predictions. Allow them to vary initial probabilities to see how the future changes. Allow them to go back and modify some of their estimates Create the cross impact model using the approach by Turoff (logistic, Fermi Dirac equations). This provides a scale that changes nonlinear probabilities (0 to 1) to a linear influence factor (Cij) between each pair of events (plus to minus infinity). When participants are satisfied with their individual model, utilize the internal linear influencing factors (Cij) to create a group model.

Cross Impact is the one of the most challenging areas of interface design today as it is the concept of allowing users to build their own models without having to program the model Cross Impact Analysis.

Cross Impact Modeling Creating Scenarios

System Functions Participants Responses System Actions on Responses
Analyze the internal parameters to show people which of the relationships between which events show the most disagreement among the group. Ask for comments on these combinations from those who have inconsistent or extreme views. View these comments and changes to some of their original estimations. Create a model of interacting scenarios by voting on where to stop the integration of the events in process that can turn all the events into one scenario. When no more changes are being made, use Interpretive Structural Modeling to generate a set of macro scenarios collecting individual events that are tightly coupled into a set of scenarios that interact. Requires human monitor to know when to trigger the scenario creation part.


Delphi has been in active use since its invention in the 1960's. Unfortunately, there are probably more examples of unsuccessful Delphi exercises than successful ones in that it sounds like a simple process – it is anything but that. There is a lot of effort and careful planning to do a successful one with a quality group of participants. Some of the best Delphis over the years were unfortunately proprietary and were never published. There have also been some very interesting controlled experiments. One experiment shows that online Delphi exercises are more productive with respect to improvisation and creativity that exhibits novelty is than unstructured online discussions without voting[3][4]. Another shows that the only time middle managers in a major corporation are willing to discuss prior past decisions as possible mistakes in planning the future of the company is when they can be anonymous in an online discussion rather than using their real name[5].


  1. Turoff, M. (1972) An Alternative Approach to Cross Impact Analysis, Technological Forecasting and Social Change 3, 309-339, also in the Delphi Method book
  2. White, Connie, Murray Turoff, Bartel Van de Walle, A Dynamic Delphi Process Utilizing a Modified Thurstone Scaling Method: Collaborative Judgment in Emergency Response, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press
  3. Cho, H.K. & Turoff, M., “Delphi Structure and Group Size in Asynchronous Computer-Mediated Communications,” Proceedings of the Americas Conference on Information Systems, Tampa, August 2003
  4. Cho, H. K. (2004) The effect of Delphi structure on small and medium-sized asynchronous groups, Ph. D. Dissertation, January 2004, New Jersey Institute of Technology, Information Systems Department. [1]
  5. Hiltz, S. R., Turoff, Murray, & Johnson, K., (1989), Experiments in Group Decision Making, 3: Disinhibition, Deindividuation, and Group Process in Pen Name and Real Name Computer Conferences, Decision Support Systems, 5, 217-232. Full report and other related experimental reports on the NJIT Library archive website at the end of the reference list.

See also

Delphi survey

Personal tools