National Institute of
General Medical Sciences

NIGMS-funded training programs must conduct ongoing program evaluations to monitor the effectiveness of their training and mentoring activities in meeting the stated goals. The following evaluation resources include guides, reports, tools and relevant literature on various aspects of evaluation. This is not a comprehensive guide to the field of evaluation or to developing an evaluation. The resources listed are free and publicly accessible, as of the time this site was developed.

The content within these evaluation resources is neither created nor endorsed by NIGMS; the listing is intended to serve as potential resources for programs as they develop and refine their evaluation plans. NIGMS isn't responsible for the availability or content of these external sites, nor does it endorse, warrant, or guarantee the products, services, or information described or offered at these sites.

The first section provides general resources on evaluation, and the second section provides resources on STEM education and research training evaluation​.

The final sections provide answers to evaluation-related frequently asked questions (FAQs) received by NIGMS program staff​​​​​.

General Evaluation Resources:

Free Resources:

• Evaluation 101 https://sites.google.com/view/evaluation101/resources?authuser=0
  An index of resources to support program evaluation planning and implementation needs.
• W.K. Kellogg Foundation Evaluation Handbook https://wkkf.issuelab.org/resource/the-step-by-step-guide-to-evaluation-how-to-become-savvy-evaluation-consumers-4.html​​
  A handbook that provides guidance for conducting and managing evaluations.
• Better Evaluation https://www.betterevaluation.org/
  The BetterEvaluation website has over 1,600 resources on evaluation tasks and options, and more are added frequently. The resources include overviews, guides, example frameworks and questions, tools or toolkits, and discussion papers.
• Evaluation Checklist Project https://wmich.edu/evaluation/checklists
  This website provides an evaluation checklist, organized by evaluation task areas, that can be used when designing and implementing an evaluation. The site also provides various guides and literature.
• The 2002 User-Friendly Handbook for Project Evaluation https://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf
  The National Science Foundation (NSF)'s 2002 user-friendly handbook for project evaluation. This guide discusses evaluation objectives and types of evaluations. It provides an overview of quantitative and qualitative data collection methods, as well as strategies that address culturally responsive evaluation.
• Survey Methods for Sensitive Topics https://graemeblair.com/papers/sensitive.pdf
  Outlines four survey techniques to protect sensitive topics.
• Introduction to Impact Evaluation https://www.interaction.org/wp-content/uploads/2019/03/1-Introduction-to-Impact-Evaluation.pdf
  This resource provides an introduction to impact evaluation, and addresses questions to support management, program and monitoring and evaluation (M&E) staff in international non-governmental organizations (NGOs) to plan, design, manage, conduct and use impact evaluations.
• 'A balancing act': Resolving multiple stakeholder interests in program evaluation https://www.anneconsulting.com.au/files/4813/9046/1077/balancing_act.pdf
  ​This paper discusses negotiation, an essential component to the planning stage of an evaluation, and strategic steps to be taken early in the evaluation process to ensure consensus is developed in stakeholder expectations regarding methodology and outcomes from the evaluation.
• Likert-type Scale Response Anchors (Vagias, Wade M. (2006)) http://media.clemson.edu/cbshs/prtm/research/resources-for-research-page-2/Vagias-Likert-Type-Scale-Response-Anchors.pdf
  Provides examples of Likert-type scales.
• Cognitive Interviewing — A "How To" Guide https://www.hkr.se/contentassets/9ed7b1b3997e4bf4baa8d4eceed5cd87/gordonwillis.pdf
  Guide outlines strategies to implement feedback process to build better surveys. To be used for questionnaire development and testing.
• Program Evaluation, Professional Development, and Research Mentoring Resources [PDF] https://www.nigms.nih.gov/training/Documents/T32-Program-Eval-Prof-Dev-and-Mentoring.pdf
  An annotated bibliography that may serve as a starting place to explore the evidence and research on several aspects of graduate training programs, specifically around program evaluation, evaluation in graduate training, graduate student professional development, research mentoring with a focus on diversity and inclusion.

STEM Education and Research Training Evaluation Resources:

Free Resources:

• Informal Science
  https://www.informalscience.org/projects/evaluation
  This website provides guides, plans, tools for STEM evaluation.
• Undergraduate Research Student Self-Assessment (URSSA)
  https://www.colorado.edu/eer/research-areas/undergraduate-research/evaluation-tools-undergraduate-research-student-self
  Undergraduate Research Student Self-Assessment (URSSA) is an online survey instrument for programs to use in assessing the student outcomes of undergraduate research.
• ​Entering Research Learning Assessment (ERLA)
  https://wiscience.wisc.edu/2019/07/10/entering-research-learning-assessment-erla/
  The Entering Research Learning Assessment (ERLA) provides research trainee and mentor surveys to assess learning in seven areas of trainee development.
• NRMN Measures Library
  https://projects.wcer.wisc.edu/materials-library/
  The National Research Mentoring Network (NRMN) measurements library provides examples of survey items, scales, and other types of measures that program directors and evaluators may find useful when assessing the efficacy of interventions in the STEM fields.
• Alliances for Graduate Education and Professoriate (AGEP)
  https://www.nsfagep.org/evaluation-resources/
  This website has a number of guides, tools, reports, and other resources on evaluation.
• The STEM Education and Training Topical Interest Group (STEM TIG)
  https://comm.eval.org/stemeducationandtraining/home
  Provides a professional home for evaluators who are looking for a community around STEM education evaluation.
• Mentor Evaluation: Training Assessment Tools
  ​https://ictr.wisc.edu/mentoring/mentor-evaluation-training-assessment-tools/
  Provides example survey instruments of tools to evaluate the effectiveness of research mentor training sessions.
• ​Diversity Program Consortium Hallmarks of Success
  ​https://nigms.nih.gov/training/dpc/Pages/success.aspx
  The Diversity Program Consortium identified key elements at each stage of the biomedical research career trajectory that lead to the development of a successful biomedical research career and, ultimately, success as an NIH-funded investigator. These "Hallmarks of Success" guided the Consortium-wide evaluation and span the individual student, faculty/mentor, and institutional levels, and may serve as an example for others undertaking an evaluation.
• Understanding Interventions Evaluation Measures
  ​https://www.uiindex.org/search/index?in_additional_index_resource=4
  A searchable database resource for evaluation resources.
• TWD Webinar on Program Assessment and Evaluation
  ​https://www.nigms.nih.gov/training/Pages/TWD-Webinars.aspx#ProgAsmtEval
  In this webinar, TWD program staff and representatives from the National Research Mentoring Network (NRMN) discuss ways to advance assessment and evaluation efforts in an efficient, evidence-informed, collected and coordinated manner.

Relevant Literature:

• Entering Research Learning Assessment (ERLA): Validity Evidence for an Instrument to Measure Undergraduate and Graduate Research Trainee Development. Butz A, Branchaw J. (2020). CBE — Life Sciences Education, 19(2), p.ar18. https://www.lifescied.org/doi/10.1187/cbe.19-07-0146
• Program evaluation practices and the training of PhD students in STEM. Reeves PM, Claydon J, Davenport GA. (2022). Studies in Graduate and Postdoctoral Education, Vol. 13 No. 2, pp. 109-131. https://doi.org/10.1108/SGPE-04-2021-0029
• The role of scientific communication in predicting science identity and research career intention. Cameron C, Lee HY, Anderson CB, Trachtenberg J, Chang S. (2020). PLOS ONE, 15(2): e0228197. https://doi.org/10.1371/journal.pone.0228197
• Perspective: Adopting an Asset Bundle Model to Support and Advance Minority Students' Careers in Academic Medicine and the Scientific Pipeline. Johnson J, Bozeman B. (2012). Academic Medicine: Journal of the Association of American Medical Colleges, 87. https://doi.org/10.1097/ACM.0b013e31826d5a8d
• Where Do Our Graduates Go? A Tool Kit for Tracking Career Outcomes of Biomedical PhD Students and Postdoctoral Scholars. Silva E, Mejía A, Watkins E. (2019). CBE — Life Sciences Education, 18(4), p.le3. https://doi.org/10.1187/cbe.19-08-0150
• Questioning Assumptions. Poodry C, Asai D. (2018). CBE — Life Sciences Education, 17(3), p.es7. https://doi.org/10.1187/cbe.18-02-0024
• The Mentoring Competency Assessment: validation of a new instrument to evaluate skills of research mentors. Fleming M, House S, Hanson VS, Yu L, Garbutt J, McGee R, Kroenke K, Abedin Z, Rubio DM. (2013). Academic Medicine: Journal of the Association of American Medical Colleges, 88(7), 1002–1008. https://doi.org/10.1097/ACM.0b013e318295e298
• Competency-based assessment for the training of PhD students and early-career scientists. Verderame MF, Freedman VH, Kozlowski LM, McCormack WT. eLife. 2018 May 31;7:e34801. https://doi.org/10.7554%2FeLife.34801

Relevant Journals​:

• American Journal of Evaluation: https://www.eval.org/Publications/Publications/American-Journal-of-Evaluation
• BMC Proceedings: https://bmcproc.biomedcentral.com/
• CBE — Life Sciences Education: https://www.lifescied.org/
• Journal of Research Administration: https://www.srainternational.org/resources/journal
• New Directions for Evaluation: https://www.eval.org/Publications/Publications/New-Directions-For-Evaluation
• PLOS One: https://journals.plos.org/plosone/
• Research in Higher Education: https://www.springer.com/journal/11162