Models of Infectious Disease Study (MIDAS) Process Evaluation Report Summary

Introduction

The Models of Infectious Disease Agent Study (MIDAS) is a consortium of researchers who develop mathematical and computational tools to assist policymakers and public health professionals in preparing the nation for outbreaks of infectious diseases. The program is managed by the National Institute of General Medical Sciences (NIGMS) and is administered through the cooperative agreement funding mechanism. The MIDAS network consists of seven Research Groups (three awards made in 2004 and four in 2005) and an Informatics Group (award made in 2004); the network is overseen by a Steering Committee which includes all Principal Investigators, representatives from the NIH, and several external experts.

The 2004 cohort of the MIDAS scientists is facing competitive renewal in 2008. In preparation for the next solicitation, the NIGMS contracted with Abt Associates to conduct an independent process evaluation of MIDAS. The evaluation focused on three aspects of the program: adequacy and appropriateness of policies and infrastructure; effectiveness of internal and external collaborations and communications; and perceived value of outputs. Data for the evaluation were gathered by interviewing 35 individuals, including all Principal Investigators, several collaborators, federal and state government staff (participating in MIDAS and not), and an outside academic scientist. In addition to the interviews, Abt reviewed extant program materials such as publications, the MIDAS web portal, solicitations, and policy documents. Findings from the study were reported to the NIH staff and revisions made based on their input. This document is the report summary.

Findings

Policies and infrastructure

The MIDAS project was considered a success by respondents, in no small part due to its management by an exceptionally dedicated, diplomatic, and knowledgeable staff at the NIH. The MIDAS Scientific Director Dr. Eckstrand, in particular, was able to facilitate productive collaborations despite some difficult challenges. Indeed, close engagement by the NIH was cited by many individuals as one of the most valuable aspects of MIDAS and as an attribute that set the program apart from more traditional, single-investigator grants. One suggestion for improvement in the management style of the program was to provide more explicit guidance to the network on what level of contribution to the policy making process the groups are required to make and what network activities are optional.

Although Abt obtained uniformly positive feedback about the NIH, respondents’ opinions about the Steering Committee diverged substantially: some interviewees viewed the Steering Committee as ineffectual, while others as indispensable. After carefully considering all input, Abt concluded that while changes in the composition of the Steering Committee could be made (for example, including more end-users), this system of governance should be maintained. Steering Committee members interviewed appeared to be extremely knowledgeable and able to provide keen and valuable insights about the network. The evaluators felt that a Steering Committee composed of such highly qualified and resourceful individuals goes far to enhance the credibility of MIDAS.

When asked to comment on the adequacy of funding support, some of the PIs and about half of collaborators noted that the amount was insufficient for implementing both the proposed goals and the policyrelated research. The shortfall identified, however, was relatively small, on the order of $100,000 a year (for example, "sufficient to hire a research fellow). Surprisingly, PIs do not take full advantage of the opportunity to apply for supplemental funds through the NIGMS, despite a nominal application process. The five-year duration of the award was considered appropriate.

Based on the interviews and on the analysis of the portal data, Abt concluded that network participants interact on a very frequent basis: in 2006, for example, 25 joint activities were held, excepting informal phone calls and email exchanges. The formal activities, for which summary notes and presentations are posted on the portal, included network meetings, videoconferences, Steering Committee meetings, government briefings, and consultations. For the most part, respondents appeared to enjoy communal events, albeit consultations were identified as least beneficial. Respondents lacked clarity whether attending formal events was required or optional.

In the course of the study Abt observed that the MIDAS researchers, in particular those involved in the influenza project, found it difficult to balance requests for policy-related work with pursuing their own research goals. The conflict was most pronounced when the researchers were asked by the government to perform work that they did not consider scientifically meritorious. A suggestion for making the most efficient use of the valuable MIDAS tools that emerged early in the course of the study was to shift the mission of the Informatics Group to providing computational support and facilities for MIDAS researchers and transforming the models into flexible and usable tools. With substantial support for these important but unexciting obligations, the MIDAS scientists would be able to focus primarily on novel, more creative work. The scientists, however, should be required, perhaps as a condition for an award, to provide guidance and supervision to this proposed new Informatics Group.

Collaborations and communication

According to some respondents, the Informatics Group has fallen short of expectations. Asymmetry in the distribution of MIDAS funds (the Informatics Group received more money than each of the research groups), preference by the modelers for distributed over centralized computing, reluctance of the researchers to have their models validated by an outside group, misunderstandings, and personality conflicts all contributed to its under-utilization. That said, the evaluators were able to find ample evidence of productive collaborations betweenthe Informatics Group and the Research Groups; virtually every MIDAS participant could name a joint project with the Informatics Group and some of these collaborations resulted in publications, tools, and access to external computing resources. Most respondents felt that the scope of the Informatics Group should be reduced to performing basic tasks, such as organizing of meetings, cataloguing datasets, and maintaining of a website. Abt suggests that the Informatics Group be restructured to deliver these services in addition to performing thetasks requested by policymakers and perfecting the tools developed by research groups. Restructuring the Informatics Group in this fashion would enhance its utility both to the network and to the government clients. Funding for any activities beyond these could be provided as the need arises conditional on the approval by the Steering Committee. Also, the NIH might consider changing the name for this new entity, to a "MIDAS support center," for example, to better communicate its role and status within the network.

Currently, the MIDAS network has a policy that models and datasets have to be deposited on the portal. Abt observed that the MIDAS participants were reluctant to share their resources. They were concerned that the models and systems under study are sufficiently complex that it is critical that modeling exercises involve collaboration between policymakers and model developers. Failure to use models appropriately could well lead to inappropriate government policies. Abt evaluators also perceived a certain lack of trust within the network regarding the intellectual property produced by the groups, although no specific examples of violations were offered during the interviews. Given these findings, Abt suggests that data sharing rules be revised to make them more practical and acceptable to the MIDAS participants.

The utility of the MIDAS portal in its current format appeared to be limited (80% of respondents said that they rarely use the portal and 15% that they never do). Abt reviewed the contents of the portal and identified areas for improvement: for example, search engines yield confusing results and some of the information meant for internal use by the network participants seemed simplistic. In the opinion of the evaluation team, the portal would be most effective as MIDAS advertising, educational, and library resource aimed at outside communities. It could be enhanced through dynamic simulations of influenza models with clear descriptions of assumptions and limitations, to better illustrate the MIDAS network capability. Tools that the MIDAS participants have already developed and that might be of use to others, such as synthetic populations and 1918 influenza database, should be well annotated and made easily accessible through the portal. Additional resources generated by MIDAS could be shared through the portal once they have been perfected by the Informatics Group.

Most respondents agreed that collaboration on the influenza modeling project resulted in scientific and policy contributions that exceeded the capabilities of individual groups, although the evaluators were repeatedly told of intense competition among some of the modelers. It appeared that the level of collaboration between the cohorts varied. While the groups of the first cohort partnered on the influenza project, most groups of the second cohort functioned more autonomously. Abt found that while PIs from the second cohort are interested in collaboration in principle, their expertise has not been engaged by existing joint projects. To strengthen collaborations, Abt suggests that the Steering Committee agree upon a specific research project to which new groups can contribute their diverse expertise.

While MIDAS has clearly established links to several of the relevant offices within the government, the evaluators observed that some level of tension remained between the network and the CDC. Since the engagement of the CDC is critical for a program such as MIDAS, Abt suggests issuing a joint solicitation with this agency to help further facilitate the relationship. This more formal engagement would not only be a gesture of goodwill by the NIH, it could also simplify data exchange processes and open new venues for collaboration. Not the least, a joint solicitation would emphasize a more applied nature of the MIDAS network to the scientific community.

The language and conceptual gaps that remain between the academic and policy communities appeared to be a source of concern to the scientists and to the government officials alike. The MIDAS network can make an important and lasting contribution by bridging traditionally isolated communities through training researchers versed in both modeling and policy/public health. The NIGMS can test this idea by awarding a small number of MIDAS postdoctoral fellowships to scientists with backgrounds in mathematical modeling who are interested in policy or public health training. A joint solicitation with the CDC would be an asset if such training modules were established.

Outputs and outreach

Abt concluded that in the few years since its launch, MIDAS has made impressive progress towards achieving its goals. At present, the network includes close to 60 participants affiliated with a variety of academic, government, and commercial organizations in the United States and abroad ( Figure 1). The program has clearly recruited talented and productive researchers: since 2004, the MIDAS groups published 38 papers, several in the most prestigious academic journals Science and Nature. MIDAS research is well regarded by the academic community, as evidenced by a large number of citations of articles authored by members of the network (almost 450, in total) and by the quantitative measures of journal influence. Importantly, in addition to this formidable academic contribution, work of the MIDAS scientists formed the basis of key national policy documents on the mitigation of an influenza outbreak and established a link between the network and the policy community. This collaboration between the government and the modelers is an important achievement as it provides a clear demonstration that modeling can be effectively used as a tool in policy and planning.

Most respondents expressed very positive views about the quality of MIDAS research. Some interviewees regarded the influenza work as the most important MIDAS achievement, while others suggested that the lesser known theoretical work represented a more significant intellectual contribution. The evaluators also heard some dissenting voices, who expressed their reservations about the utility of the influenza models. This view was especially prevalent in respondents representing the public health community. These individuals arguedthat since the models did not incorporate some key parameters related to the biology of the virus (e.g., mechanisms of transmissionand possibility of drug resistance) and to the logistics of the intervention process, the guidance that they yielded was unrealistic.Limitations of the models were also noted in a report by a committee of experts convened by the Institute of Medicine.

Several respondents spoke of dangers inherent in the use of modeling in response planning. Both the scientists and the governmentofficials expressed a concern that the MIDAS tools tended to be excessively relied upon by the government in crafting policy recommendations. Language and conceptual gaps that remain between the scientists and the government decision makers contribute to the challenge of translating modeling results into sound policy guidance.

Recommendations

Based on the evaluation findings Abt Associates make several suggestions that might strengthen the MIDAS program and better align it with the stated goals:

  1. Retain the Steering Committee, though perhaps with some changes in composition, as the MIDAS governing body;
  2. Revise the charge of the Informatics Group in order to facilitate transfer of information to policy and public health communities, so that research groups can concentrate on scientific work;
  3. Convert the portal into an advertising, educational, and library tool to increase its utility;
  4. Revise data sharing polices to make them more practical and acceptable to the network;
  5. Define a common research topic, amenable to diverse scientific approaches of the MIDAS participants, to strengthen collaborationbetween research groups, of the second cohort in particular;
  6. Issue a joint solicitation with the CDC to formalize partnership with this key agency; and
  7. Develop an interdisciplinary training component to help bridge language and conceptual barriers between the academic and the policy communities.
Source: Interviews with MIDAS Principal Investigators and collaborators (summer of 2007) with input from the NIH staff
Source: Interviews with MIDAS Principal Investigators and collaborators (summer of 2007) with input from the NIH staff