About this Database Background Methods Rating Scales Choosing Your Strategy Sources Health Conditions & Diseases Acknowledgements Contact Us Disclaimer Search Policies & Programs

hints
Display All Policies & Programs

Evidence of Effectiveness

“Evidence of effectiveness” can mean different things to different people. Our approach to assessing evidence combines what we know from scientific study and the observations of unbiased experts.

Evidence Rating: Search, Selection, and Assessment

We begin with a broad orientation search to define each strategy and identify appropriate search terms. We then conduct targeted literature searches, focusing first on systematic reviews and peer reviewed articles, then on selected sources of grey (unpublished) literature and the findings of relevant, reputable organizations that assess policy and program effectiveness (rating organizations). All searches are conducted electronically. The sources we search vary by health factor and strategy.

Retrieved articles are screened by date, relevance to topic of interest, applicability of findings, study type, and impartiality of author(s). We retain the most relevant, recent, rigorous reviews and articles for consideration in evidence rating. Ratings are assigned based on two analysts’ assessments of the strength of the overall body of evidence (e.g., type, quality, and number of studies, consistency of findings, etc.). We place the most weight on the findings of studies with designs that demonstrate causality; we consider study quality in conjunction with design. External content experts also review ratings.

Evidence Rating: Guidelines

Rating Evidence Criteria: Amount & Type Evidence Criteria: Quality of Evidence
Scientifically Supported
  • 1 or more systematic review(s), or at least:
  • 3 experimental studies, or
  • 3 quasi-experimental studies with matched concurrent comparisons
Studies have:
  • Strong designs
  • Statistically significant positive findings
Some Evidence
  • 1 or more systematic review(s), or at least:
  • 2 experimental studies, or
  • 2 quasi-experimental┬ástudies with matched concurrent comparisons, or
  • 3 studies with unmatched comparisons or pre-post measures

Studies have statistically significant positive findings

Compared to 'Scientifically Supported', studies have:

  • Less rigorous designs
  • Limited effect(s)
Expert Opinion
  • Generally no more than 1 experimental or quasi-experimental study with a matched concurrent comparison, or
  • 2 or fewer studies with unmatched comparisons or pre-post measures
  • Expert recommendation supported by theory, but study limited
  • Study quality varies, but is often low
  • Study findings vary, but are often inconclusive
Insufficient Evidence
  • Generally no more than 1 experimental or quasi-experimental study with a matched concurrent comparison, or
  • 2 or fewer studies with unmatched comparisons or pre-post measures
  • Study quality varies, but is often low
  • Study findings vary, but are often inconclusive
Mixed Evidence
  • 1 or more systematic review(s), or at least:
  • 2 experimental studies, or
  • 2 quasi-experimental studies with matched concurrent comparisons, or
  • 3 studies with unmatched comparisons or pre-post measures
  • Studies have statistically significant findings
  • Body of evidence inconclusive, or
  • Body of evidence mixed leaning negative
Evidence of Ineffectiveness
  • 1 or more systematic review(s), or at least:
  • 3 experimental studies, or
  • 3 quasi-experimental studies with matched concurrent comparisons
Studies have:
  • Strong designs
  • Significant negative or ineffective findings, or
  • Strong evidence of harm