Evaluation Reports

Current Reports

NIGMS’s Institutional Clinical Postdoctoral Research Training T32 Grants

Evaluation of Postdoctoral NRSA T32 Institutional Training Grant Program [PDF]

June 27, 2018

Program Type: T32

Pre Evaluation FOA: T32 Training Grants

Evaluation of Postdoctoral NRSA T32 Institutional Training Grant Program, PDF  
  1. NIGMS continue its support of the postdoctoral NRSA CPRT T32 programs and re-assess the emphasis of the programs in regard to changes in clinical research and practice, as well as changes in financial obstacles faced by clinician scientists that have occurred since the inception of the programs (e.g., income differential compared to clinical pursuits).
  2. NIGMS should define clear goals for its clinically focused research training programs (CPRT T32, MSTP, K08/K23) and develop metrics than can be routinely collected by each CPRT program and collectively by NIGMS for all CPRT programs. Such data could provide a basis for measuring success and continuing enhancement for each program and for CPRT programs overall. The committee felt that, if required, support for such data collection would be beneficial for all institutional and individual training grant mechanisms.
  3. NIGMS should reexamine the focus research areas of Medicine and Clinical Research of these programs to ensure they are best serving the clinical research workforce needs such as: exploring Medical Genetics, identifying NHGRI areas of commonality with the possibility of co-funding or the transfer of medical genetics training to NHGRI. Broaden Trauma, Burn, and Injury to include emergency and critical care medicine. Examine Clinical Pharmacology to see what it encompasses today (for example, pharmacogenetics, systems biology, mathematical models, big data), current workforce needs, and how this fits within the NIH. Consider incorporating pain medicine more clearly within the Anesthesiology focus area.
  4. Examine the feasibility of expanding loan repayment eligibility for clinical investigators in the 4 NIGMS focus areas, once updated.

Findings from this evaluation were presented during the May 2018 NIGMS Council Meeting

During NIGMS Advisory Council meeting in September 2018, the Training and Workforce Diversity Division Director, outlined steps that to address the recommendations

  • Recommendation 1 Response: NIGMS will continue to support and monitor the programs by conducting portfolio analyses, attending relevant clinical research meetings, interacting with relevant professional societies. NIGMS is aware of the financial obstacles faced by clinician scientists and will continue to monitor income differential of clinical research careers compared to clinical pursuits and explore solutions to retain clinician-scientists.
  • Recommendation 2 Response​: NIGMS has posted goals of the programs on the NIGMS postdoctoral T32 website. To enhance data collection, NIGMS formed collaborations with:
    • The Census Bureau to analyze student data longitudinally (collaboration through 2019).
    • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.
  • Recommendation 3 Response: NIGMS conducted analyses and outreach in each of the areas to ensure that the training was in alignment with current trends in clinical research. For example:
    • Medical Genetics – NIGMS and NHGRI communicate regarding funding of clinical postdoctoral trainees to ensure there is not overlap.
    • Trauma, Burn, and Injury – NIGMS changed the name to “Injury and Critical Illness” to reflect the complexity of the portfolio.
    • Clinical Pharmacology – NIGMS is in communication with the research community and professional societies to ensure that the training is in alignment with the research workforce needs.
    • Anesthesiology – the NIGMS website was updated to reflect the breadth of the portfolio, including pain research.
  • Recommendation 4 Response: The loan repayment eligibility criteria are controlled by Congress and outside of NIGMS’ authority. NIGMS reached out to the Director of the Loan Repayment Office to pass on the concerns of the committee.

June 15, 2018

Program Type: R25

Pre Evaluation FOA: PAR-17-209

Evaluation of the NIGMS R25 Bridges to the Doctorate Program, PDF  

Conclusion:

The absence of high-quality trainee data made it essentially impossible to critically evaluate the Bridges to Doctorate program.

Recommendations:

  1. To benchmark the NIGMS B2D program against the NIGMS Post Baccalaureate Research Education Program (PREP) and NSF Louis Stokes Alliance for Minority Participation (LSAMP) Bridge to the Doctorate programs to identify the most successful and cost-effective mechanism(s) to increase underrepresented Ph.D. graduates, with the goal of reducing, remodeling, or phasing out the NIGMS B2D program.
  2. To increase resource flow toward (centralized) data collection mechanisms that improve the longitudinal tracking of trainees and trainee outcomes.

Additional Recommendations:

  1. To increase resource flow toward (centralized) data collection mechanisms that improve the longitudinal tracking of trainees and trainee outcomes.
  2. Conceive strategies to enhance research activity through partnership.

The evaluation was presented at the May 2018 NAGMS Advisory Council by a member of the external evaluating committee.

A presentation of the NIGMS response to the B2D Evaluation was presented to the September 2018 NAGMS Advisory Council by the NIGMS Training, Workforce Development and Diversity Division Director. The main recommendation responses are as follows:

  • Recommendation 1 Response: The NIGMS Training, Workforce Development and Diversity Division Director, the Undergraduate and Postdoctoral Branch Chief, and Program Staff met with National Science Foundation (NSF) officials overseeing the Louis Stokes Alliances for Minority Participation (LSAMP) Program and discussed the potential for the benchmark comparison, but uniformly arrived at the consensus that the NIGMS and NSF programs are distinct and therefore not comparable. The recent program evaluations based on progress reports show that PREP program have similar outcomes for transitioning into and completing Ph.D. programs.
  • Recommendation 2 Response: NIGMS has since formed collaborations with the Census Bureau to analyze student data longitudinally and with the NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.
  • Additional Recommendation Response:​ Based on the outcome evaluation and analysis of successful long-standing programs, the funding opportunity announcement has been subsequently revised and reissued (PAR-19-300). Relevant outcome evaluation features include:
    • Stating the proximal and long-term goals of the program. The proximal goal states that the objective is to develop a diverse pool of scientists earning a Ph.D., who have the skills to successfully transition into careers in the biomedical research workforce.
    • Emphasizing strong partnerships to enhance bridging, reduce the time to degree, and build research capacity at the master’s degree granting institution (e.g., with effective skill-building activities, course credit articulation agreements, and shared research resources).
    • Focusing on training and mentoring interventions before and after the bridge to ensure a smooth transition and increase persistence.
    • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
    • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
  • The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The Bridges to the Doctorate Research Training program supports individuals as they make the transition from master’s degree granting institutions to research-intensive to Ph.D. degree granting institutions.

Models of Infectious Disease Agent Study (MIDAS) Program

MIDAS Program Evaluation Report (STPI) [PDF]

September 2016

Program Type: U01

Pre Evaluation FOA: RFA-GM-14007

MIDAS Program Evaluation Report (STPI), PDF  
  1. If additional analysis of MIDAS outreach is desired in the future, NIGMS should mandate that awardees give additional care to the outreach activities sections of their annual reports.
  2. Many investigators stated that they had proposed outreach programs, especially short courses and conferences, which were not implemented due to limited funds, and that cuts in funding would affect outreach activities first.
  3. To build on MIDAS’ record of policy engagement and improve its impact on public policy decisions, policy makers and researchers suggested that MIDAS support further engagement including, creative initiatives that allow researchers to work more closely within government offices.
  4. Data access policies and data sharing practices presented significant barriers for some MIDAS investigators who wanted to provide policy relevant modeling information to federal officials during disease outbreaks.
  5. It is important to develop clear standards and shared expectations to foster successful collaborations among an array of modelers and policy makers. Consensus data sharing agreements, standards, and policies are necessary to expedite time-sensitive collaborations and enhance the utility of modeling outputs for policy decisions.
  6. The complexity and diversity of MIDAS infrastructure presents challenges in meeting the programmatic goals of sharing and dissemination of MIDAS supported models and tools.
  7. NIGMS may need to develop sharing metrics that are appropriate for the specific type of resource.
  8. NIGMS may need to reconsider its goals for sharing and dissemination of MIDAS resources given the current issues and future shifts in information technology, data security and public policy. MIDAS investigators suggested the following to enhance relationships with policy makers:
    • Deployment of modeling research for epidemic support should be an “end to-end data to decision support” paradigm.
    • Modelers should become an active part of public health teams to provide data-driven input regarding policy decisions at all times.
    • MIDAS could have a more formal internal system to identify and match policy makers with the appropriate researchers given the nature of expertise and type of model desired.
  • Following this report, NIGMS eliminated all U grants except for a single U24 grant that funds a Coordination Center, which acts as a focal point for collaboration and training, testing and dissemination of MIDAS research projects. See RFA-18-003.
  • This funding opportunity provides guidance on the infrastructure and resources for evaluating and disseminating software, collecting and managing data relevant to MIDAS systems.
  • MIDAS applicants must now provide support for dissemination and outreach and providing logistical support for MIDAS activities such as scientific and Educational coordination and outreach.

Institutional Research and Academic Career Development Award (IRACDA) (K12)

Institutional Research and Academic Career Development Awards (IRACDA) (K12) Outcomes Assessment [PDF]

June 21, 2016

Program Type: K12

Pre Evaluation FOA: PAR-13-290

Institutional Research and Academic Career Development Awards (IRACDA) (K12) Outcomes Assessment, PDF  
  1. IRACDA scholars transition into academic careers at a high rate (73%) and hold academic positions across a broad range of institution types (2-year, 4-year, research intensive, minority-serving) throughout the country.
  2. IRACDA scholars contribute the diversity of the workforce with 63% female, 17% Hispanic and 19% African-American participants.
  3. IRACDA alumni have success rates in securing research funding similar to a matched comparator set of F32 awardee recipients.

Findings of the report were presented during the May 2016 NIGMS Council, and published in a June 2016 Feedback Loop Post.

  • Based on the outcome evaluation and analysis of successful long-standing programs, the funding opportunity announcement has been subsequently revised and reissued (PAR-18-366). Relevant outcome evaluation features include:
    • Stating the proximal and long-term goals of the program. The proximal goal states that the Overarching Objective of the IRACDA program is to develop a diverse pool of well-trained biomedical scientists, who have the technical (e.g., appropriate methods, technologies, and quantitative/computational approaches), operational (e.g., independent knowledge acquisition, rigorous experimental design, and interpretation of data) and professional (e.g. management, leadership, communication, and teamwork) skills necessary to conduct rigorous and reproducible research, and to transition successfully into independent academic careers in the biomedical research workforce.
    • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
    • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
  • The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The IRACDA program supports individuals at the early stage of the postdoctoral career.
  • There were some challenges during the analysis, including verifying workforce information. To address this, NIGMS has since formed collaborations with:
    • The Census Bureau to analyze student data longitudinally.
    • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

May 5, 2016

Program Type: R24

Pre Evaluation FOA: PAR-13-324

Outcomes Evaluation NIGMS Legacy Community-Wide Scientific Resources Pilot Program, PDF  
  1. Strengthen the language in future FOAs, particularly pilot programs, to further emphasize the importance of metrics and evaluation.
  2. Researchers who develop resources as part of a grant should be encouraged to find ways to track users of the resource and should encourage the citation of the resource by its users/user community in publications, track the number of publications/citations in which the resource is referenced, and find additional methods of measuring productivity and impact.
  3. Any concerns noted by reviewers regarding lack of adequate evaluation plans or insufficient metrics should be addressed with PIs upon award to ensure that progress reports contain the required data and that the deficiencies have been addressed and documented.
  4. Program officers (POs) should review progress reports to ensure that the required information is included or that there is an explanation why it is not included and should address any concerns with PIs.
  5. Consider developing a reporting template or standard reporting format for capturing utilization rates in progress reports and monitor compliance through regular review and follow-up by POs.
  6. Encourage PIs to conduct and share results of internal evaluations.
  • To continue support of vital resources that are no longer eligible for support under their original initiatives, a separate funding opportunity announcement was created for NIGMS-sponsored "legacy" resources. See PAR-19-208.
  • NIGMS program staff continues to work with Legacy Resource Principal Investigators to include metrics for annual Research Performance Progress Report (RPPR) submissions.

May 2016

Program Type: P41

Pre Evaluation FOA: PAR-14-021

Report of the Biomedical Technology Research Resources External Review Committee, PDF  
  1. Although the panel felt the BTRR program has been successful in its mission, there was broad agreement that it could nonetheless be improved. In particular, the resources supported by the BTRRs in the aggregate should be assessed to address two shortcomings: First, redundant resources are sometimes provided by more than one BTRR. Second, some critical technologies that would meet existing needs are missing.
    • Overall size of the BTRR program is essentially fixed, however, the low rate of turnover (1 per year) prevents new technologies from being introduced.
    • More details should be available about how novel technologies resulted from the BTRRs. As such, need better documentation of patents, patent citations, and commercialization of technologies.
    • Visibility of the BTRR program should be raised by promotion and highlighting of their capabilities.
    • The panel thought the overall five-component structure for a BTRR was reasonable, but that there should be flexibility and modularity in how these components are implemented.
    • The panel envisioned three “phases” of support (initial phase supporting higher risk technology development; middle phase funding continued efforts, final phase supporting established technologies) using different grant mechanisms, perhaps something along the lines of the R21/R33 or SBIR/STTR pathways.
    • BTRRs should be better integrated with existing technology-focused R01s within NIGMS. Similarly, R21s could allow for initial support for development of more novel and risky technologies ultimately leading to new BTTR formation.
  2. The panel saw significant merit in the NIGMS BTRR program, as well as the technology-focused R01s in NIGMS. These technological advances have direct impact on biological and biomedical problems relevant to the mission of NIGMS. The panel had several ideas on how BTRRs and technology-based grants could be better integrated into NIGMS:
    • Identify the range of technology based R01s to determine new techniques that might be developed into BTRRs;
    • Improve the coordination of the review of P41s and technology-based R01s. This may include a common review panel for applications proposing research resources and technology development.
  3. Additional suggestions included:
    • To enable better comparisons, there is a need to modify the review procedures so that multiple P41 applications are reviewed as a group.
    • NIGMS might explore the joint review of their BTTR and NIBIB Biomedical Technology Resource Center [BTRC] applications, thereby enhancing coordination of the two programs and the range of centers they support.
    • More flexibility should be allowed in the types and number of projects (and components).
    • The program should recognize the fundamental differences in technologies and their ability to be disseminated.
    • The panel was split on the potential merits of a sunset policy for BTRRs. However, transitioning to other support mechanisms is appropriate for mature technologies and critical site-specific resources.

Findings of this evaluation report were presented during the May 2016 NAGMS Council meeting.

  • This report provided an initial use of comparative programs for evaluation analysis. In looking for ways to show return on investment output, NIGMS OPAE expanded network analysis for collaborative mechanisms that demonstrate connections between individuals.
  • In response to the BTRR evaluation findings, NIGMS announced September 2016 NAGMS Council Meeting, 2017 NIGMS Feedback loop post several program enhancements developed into PAR-17-316 including:
    • Emphasizing a shift from resource access to technology dissemination
    • Streamlining and simplifying program structure through consolidation of the BTRR program components from five to three: Research and Development (TR&D), Driving Biomedical Projects (DBPs), and Community Engagement (CE).
    • Utilizing Technology Development R01 and R21 mechanisms to support early-stage and validation technology projects. See PAR 17-045 and PAR-17-046.
    • Developing new BTRR feature, "Technology Development Partnerships" which will allow for more flexibility investigators who run centers to share their technologies with the community. See PAR-17-316.
  • Retirement of resources beyond year 15. Applicants must have the concept of their technology center and complete the development and dissemination within a 15-year period. See NOT-GM-18-042
  • Migration of synchrotron resources to a separate P30 mechanism.
  • Evaluation and review of the BTRR application pool through a two-stage mail and editorial panel. This allows applications to be reviewed as a whole each round, replacing site visits.
  • PAR 19-301 and PAR-19-272 have been developed to continue NIGMS support for regional and national access to resources that do not require further major R&D.
  • During the September 2019 NAGMS Council meeting, Dr. Susan Gregurick presented concept clearance for the development of the Biomedical Technology Development and Dissemination Centers (BTDD) program. This program will focus on later, iterative, and hardening stages of technology development including beta testing with driving biomedical projects in the first project-funding period. The concept has been approved by the Council.

Program Project Grants (P01s)

Project Grant (P01) Analysis

April 20, 2016

Program Type: P01

Pre Evaluation FOA: PAR-13-280

Program Project Grants (P01s)  
  1. P01s are awarded 4x as much in Total Costs compared to R01s and Multi-PI R01s, per project, this disappears.
  2. Multi-PI R01s outperform R01s and P01s in terms of the cost effectiveness of publication production.
  3. No statistically significant difference exists in the cost effectiveness of citations for these mechanisms.
  4. No statistically significant difference is present between mechanisms in terms of Relative Citation Ratio.
  5. Although both mechanisms publish in similar fields , Multi-PI R01s have more articles in the Top 1% and top 10% of cited papers than P01s.
  6. Little difference in the size of the collaborative network between P01 and Multiple R01 cohorts.

The findings of this analysis were presented during the January 2016 NAGMS Council meeting and published in an April 2016 Feedback Loop post.

  • This analysis was a primer to identify how NIGMS incentivizes team science.
  • The findings of this analysis were presented during the January 2016 NAGMS Council meeting and published in an April 2016 Feedback Loop post
  • Post-assessment, NOT-GM-17-016 encouraged those interested in collaborative research that requires multidisciplinary teams to apply to PAR-17-340 "Collaborative Program Grant for Multidisciplinary Teams" (RM1).
  • This analysis was an opportunity to clarify and communicate OPAE's role in NIGMS evaluation operations leading to the development of OPAE Standard Operating Procedures.
  • This was the first analysis to contribute to the development of cost-effective measures for evaluations; Collaborative opportunity with the NIH Library to create a standardized Extramural Investigator network analysis from Profile Person Identification (PPID)s and PubMed IDs.

Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) program

Educational Outcomes from the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) Program [PDF]

March 14, 2016

Program Type: T34

Pre Evaluation FOA: PAR-16-113

Educational Outcomes from the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) Program, PDF  
  1. Approximately 70% of recent MARC alumni have enrolled in graduate programs or earned subsequent graduate degrees in science or health, with almost one-third in recent years earning a Ph.D. or M.D.-Ph.D.
  2. The overall rate for Ph.D. attainment observed for recent MARC alumni is almost twice that seen for undergraduates supported by NIGMS supplements to enhance diversity, and four times higher than that modeled for biology baccalaureates in general.
  3. MARC alumni who matriculate in a doctorate program, two-thirds complete the Ph.D., a rate that is higher than the national average in the sciences.

The findings of this analysis were presented during the May 2016 NIGMS Council meeting videocast and published in a May 2016, NIGMS Feedback Loop post.

  • Based on the outcome evaluation and analysis of successful long-standing programs, the funding opportunity announcement has been subsequently revised and reissued (PAR-19-219). Relevant outcome evaluation features include:
    • Stating the proximal and long-term goals of the program. The proximal goal states that objective of the MARC program is to develop a diverse pool of undergraduates who complete their baccalaureate degree, and transition into and complete biomedical, research-focused higher degree programs (e.g., Ph.D. or M.D./Ph.D.).
    • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
    • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
  • The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The MARC program will support research-oriented individuals at the undergraduate level for 2-3 years.
  • There were some challenges during the analysis, including verifying workforce information. To address this, NIGMS has since formed collaborations with:
    • The Census Bureau to analyze student data longitudinally.
    • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

January 2016

Program Type: P50

Pre Evaluation FOA: PAR-14-319

Report of the National Centers for Systems Biology External Review Committee, PDF  
  1. NCSBs greatly contributed to the origins of systems biology and continue to contribute uniquely to its development. (Systems biology has grown in parallel with the centers initiative, it extends beyond the centers themselves, and it is increasing interdisciplinary.)
  2. Some reductions in overall funding to the NCSB program may be achieved by narrowing the focus and limiting the number of awards during each funding cycle.
  3. Systems biology remains potentially transformative field.
  4. NCSB program excels at integrating the diverse elements of research, training, and outreach, but should evolve in a structure appropriate to current opportunities and challenges and the pool of potential outstanding applications.
  5. Center-like funding is needed to develop the team of interdisciplinary team-orientated scientists to design, implement, and analyze screens, using computation, statistical analyses, bioinformatics, for discovery based and tool-oriented projects.
  6. NCSBs are not as cost-efficient as P01s and R01s in the production or publication of some kinds of data, and therefore may not be the best mechanism to support every kind of systems biology research.
  7. The prospect for broadening opportunities for applications of systems biology emphasizes the catalytic role NCSBs will have in influencing the rate at which the discipline penetrates the practice of science and medicine.
  8. NCSBs represent a 15-year sustained effort in training of junior investigators skilled in systems biology, and the panel recommends that this training mission be sustained and possibly enhanced by encouraging additional innovative combinations of training and outreach in the context of necessary workforce development.

The findings of this analysis were presented during the January 2016 National Advisory General Medical Sciences (NAGMS) Council Meeting).

  • The NCSB program was allowed to sunset, see NOT-GM-14-120.
  • Committee recommendations for NIGMS to encourage team science were incorporated into the design of the Collaborative Program Grant for Multidisciplinary Teams (RM1).
  • Funding announcement, PAR-17-340, for Multidisciplinary Teams (RM1) encourages applicants who propose to conduct research to address complex and challenging biomedical problems, do so through deeply integrated, multidisciplinary research teams.
  • The NCSB analysis created an opportunity to further develop standard procedure for NIGMS evaluations.

August 28, 2015

Program Type: R25

Pre Evaluation FOA: PAR-14-076

Analysis of Scholar Outcomes for the NIGMS Postbaccalaureate Research Education Program, PDF  
  1. Ph.D. Matriculation Rate: Several data sources suggest the current PREP matriculation rate into Ph.D. programs is about 65%.
  2. Ph.D. Degree Attainment: For a PREP cohort analyzed, at least 38% earned a Ph.D. (191 earned a Ph.D. out of 501 total PREP participants), at least 16% earned a M.S., 12% earned a M.D., and 7% were still in graduate training at the time of analysis; for 27% no further degree beyond the bachelors could be confirmed.
  3. The majority of PREP Ph.D. degrees were earned at research-intensive universities and medical schools.
  4. Renewal reports suggest that about 11% of the Ph.D. enrollment at PREP institutions is composed of individuals from underrepresented racial and ethnic groups (range 6-16%; 2011-2012). PREP scholars who have now completed their doctoral and postdoctoral training are predominantly entering the research workforce (43%) and science-related non-research (36%) categories.

The results were published in a September 2015 and discussed in a Feedback Loop post.

  • Based on the outcome evaluation and analysis of successful long-standing programs, the funding opportunity announcement has been subsequently revised and reissued (PAR-20-066). Relevant outcome evaluation features include:
    • Stating the proximal and long-term goals of the program. The proximal goal states that the Overarching Objective of PREP remains to develop a diverse pool of well-trained postbaccalaureate participants who will transition into and complete rigorous biomedical, research-focused doctoral degree programs (e.g., Ph.D. or M.D./Ph.D.) in biomedical fields relevant to the NIGMS mission​.
    • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
    • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
  • The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The PREP program supports individuals at the postbaccalaureate level.
  • There were some challenges during the analysis, including verifying workforce information. To address this, NIGMS has since formed collaborations with:
    • The Census Bureau to analyze student data longitudinally.
    • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

NIH-Wide Supplements to Promote Diversity in Health-Related Research (DSP)

NIGMS Analysis of Supplements to Enhance Diversity (1989-2006) [PDF]

May 28, 2015

Program Type: Supplement

Pre Evaluation FOA: PAR-12-149

NIGMS Analysis of Supplements to Enhance Diversity (1989-2006), PDF  
Archived Reports