Even with high-quality rigorous research designs, implementation, and analysis, the value of research for practice guidelines can be limited by decisions made during protocol design, study procedures, and dissemination of findings.8 This section focuses on challenges related to study populations, intervention protocols, outcomes and assessment, and linking behavior changes to health outcomes.
The USPSTF relies on clear definitions of the populations that were studied to make recommendations for asymptomatic persons and to stratify recommendations for those who are at average or high risk for particular conditions (such as cardiovascular disease). This is important because intervention effects often vary depending on the risk status of the study population. However, the characteristics of the study population relating to symptoms or risk status are often not clearly described in behavioral counseling studies. For example, the USPSTF alcohol misuse evidence review focused on interventions for non–alcohol-dependent patients with risky or harmful patterns of alcohol use. One challenge of identifying appropriate studies was that some constructed inclusion and exclusion criteria to limit the number of potential participants with alcohol dependence, whereas others did not collect or report data on whether any participants met criteria for alcohol dependence.15 The evidence review also noted that behavioral counseling trials used varying terminology to describe the included populations, further complicating considerations of population comparability.15
Failure to report the proportions of study participants with and without elevated risk factors or relevant diagnoses creates additional challenges. For example, the USPSTF evidence review on behavioral counseling to promote physical activity and a healthful diet to prevent cardiovascular disease included studies in which trial participants were unselected adults (not screened for cardiovascular disease risk factors) or were screened and did not meet criteria for hypertension, hyperlipidemia, diabetes, or cardiovascular disease. Studies with mixed populations were included if no more than 50% of trial participants had relevant diagnoses. As a result, many studies were included that enrolled and analyzed mixed groups of patients. Because researchers tend to test more intensive interventions in high-risk patients, disentangling the effects of intervention intensity from the populations’ risk status was difficult.14
Feasibility of Practice or Referral
The USPSTF behavioral counseling recommendations focus on interventions that are feasible for primary care clinicians or office staff to deliver or that are available by referral from primary care and delivered in other clinical or community settings.12
Interventions that afford flexibility in delivery and require minimal specialized training are more likely to be adaptable for delivery by primary care teams. These behavioral interventions typically comprise brief motivational messages with supportive materials offered during routine health care visits. Referred interventions are more intensive and are delivered outside of the primary care setting by persons with more extensive training and expertise. Key to judging feasibility of referral is the ease of enrollment in the intervention, perhaps through such mechanisms as direct enrollment from the provider's office by telephone or e-mail,16, 17 and lack of enrollment barriers (such as high cost or limited geographic availability). Interventions provided by professionals in settings commonly available in most clinical or community settings are also referable. The effectiveness of some community interventions, including complementary public health approaches through regulation or community-wide campaigns, is evaluated by the Community Preventive Services Task Force.18
Components and Intensity of Interventions
The USPSTF evidence reviews aim to specify the components and intensity required to achieve reductions in behavioral risk factors and associated conditions. Because most behavioral interventions include multiple components, it is often not feasible to estimate the effects of individual components. Studies that evaluate multicomponent interventions often do not adequately describe the main components or use differing terminology for similar approaches, making it difficult to assess the comparability of interventions. For example, the evidence review on diet and physical activity did not address the effectiveness of specific intervention components, which would probably be helped by more thorough and consistent reporting of counseling intervention elements.14
Intervention intensity is typically classified according to the duration and number of contacts, but there is no single consistent standard. Definitions of intensity are usually derived from the available literature and defined in an ad hoc manner for each evidence review, which limits comparability across recommendations. For example, interventions to reduce high-risk alcohol use were classified as very brief (≤5 minutes), brief (>5 to ≤15 minutes), extended (>15 minutes), brief multicontact (each contact ≤15 minutes), and extended multicontact (some contacts >15 minutes). Diet and physical activity interventions were categorized as low (≤30 minutes), medium (>30 minutes to <6 hours), or high intensity (>6 hours).
Outcomes and Assessment
Assessing Potential Adverse Effects
Adverse effects from behavioral interventions could derive from labeling; discrimination associated with screening; or the effects of the interventions themselves, such as increases in other unhealthy behaviors (for example, from alcohol misuse to illicit substance use or from weight control to disordered eating) or injuries (such as those from increased physical activity). Sufficient information about adverse effects is rarely reported for all types of preventive interventions, and the USPSTF generally considers the potential adverse effects of behavioral counseling interventions to be no greater than small.19, 20 A recent meta-analysis of physical activity trials reported an increase in risk factors (such as blood pressure or cholesterol level) for as much as 10% of study participants.21 The availability of consensus-derived measures of potential harms that can be routinely included in data collection protocols would greatly facilitate the uniform reporting of adverse effects of behavioral counseling interventions. As noted in the evidence review on alcohol misuse, potential adverse effects of screening and behavioral counseling interventions for alcohol misuse have received little attention in published studies. Investigators found no studies reporting on illegal substance use, stigma, labeling, discrimination, or interference with the physician–patient relationship.15
Behavioral Outcome Measures
Many behavioral outcomes are commonly uncovered in a systematic review of studies that focus on the same risk behavior. Although there have been efforts to standardize assessment measures, particularly in areas with substantial behavioral intervention research, such as tobacco use and cessation, more typically several valid measures are used for key behavioral outcomes. To complicate matters further, different assessment methods can be used to compute the same outcome measure. For example, with diet one can calculate the percentage of energy consumed from fats by using food-frequency questionnaires, brief dietary assessments, or 24-hour recalls.22 As evidence emerges about associations between dietary components and disease, different components are emphasized for different diseases, creating further complexity. For example, studies associating diet with incident cardiovascular disease often focus on saturated fat intake,23 whereas studies of diet and cancer focus on the percentage of calories from fat; grams of fiber; and, more recently, fruit and vegetable intake.24 Equally complex are measures of physical activity, which include self-reported surveys; job classification; motion sensors; and physiologic markers, such as doubly labeled water.25 Self-reported measures using methods as varied as detailed diaries, logs, checklists of activities, or global self-reported measures of physical activity are most commonly used.25 Efforts to pool results of behavioral counseling interventions can be hampered by noncomparability of interventions and outcomes and by varying validity of measured and reported outcomes.
The lack of standardization of behavioral measures across studies often precludes meta-analysis, making it more challenging to precisely determine net benefit, assess heterogeneity of effects by meta-regression, and provide specific recommendations for behavior change targets. For example, the evidence review on diet and physical activity noted a large variation in the type of dietary outcome reported, and it was not clear whether the included outcomes were primary or secondary outcomes or whether trials selectively reported outcomes.14 The evidence review also reported an association between the type of outcome measure and the effect size. For example, effect sizes for physical activity measured as minutes per week were larger than those in studies using other measures.14
Linking Behavior Change to Clinical Biometric Markers and Health Outcomes
The USPSTF recommendations supporting certain behavioral counseling interventions rely on evidence that changing health behavior improves health outcomes with minimal harms. Few behavioral counseling studies are designed to measure effects on health outcomes, such as death; disability; quality of life; or acute events, such as a stroke. Even the assessment of intermediate biometric risk factors, such as lipid level, blood pressure, and blood glucose level, is uncommon. In the absence of direct evidence for improvements in health outcomes, alternative indications through an indirect chain of evidence to epidemiologic and other types of studies can show that the target behavior improves health outcomes.12 These associations are often represented in the analytic framework by dotted lines between changes in health behavior and intermediate health improvements or risk factor reduction and between intermediate health improvements and reductions in morbidity or mortality (Figure 1). One of the largest challenges for the USPSTF in developing positive recommendations for behavioral counseling interventions is identifying evidence that associates behavioral outcomes (such as physical activity) with health outcomes (such as cardiovascular mortality).
This challenge is apparent in the 2 evidence reviews that serve as examples for this article. The alcohol misuse evidence review states, “The assumption underlying brief behavioral counseling interventions in primary care is that reducing overall alcohol consumption or adopting safer drinking patterns will reduce the risk for medical, social, and psychological problems.”15 The review further notes that many of the available data are from cross-sectional or cohort studies that do not provide experimental evidence. The science was better for diet and physical activity to prevent cardiovascular disease outcomes, with 16 of the 25 trials of healthful diet counseling interventions reporting 1 or more intermediate health outcomes; however, it was more limited for specific disease outcomes, with only 3 of 39 trials reporting health outcomes related to cardiovascular disease.13
1. Evidence Based Medicine Working Group. Evidence based medicine. A new approach to teaching the practice of medicine. JAMA 1992;268:2420-5. [PubMed]
2. Pope C. Resisting evidence: the study of evidence-based medicine as a contemporary social movement. Health 2003;7:267-82.
3. Levin A. The Cochrane Collaboration. Ann Intern Med 2001;135:309-12. [PubMed]
4. Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman D. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med 2010;8:24. [PMC free article][PubMed]
5. Hill J, Bullock I, Alderson P. A summary of the methods that the National Clinical Guideline Centre uses to produce clinical guidelines for the National Institute for Health and Clinical Excellence. Ann Intern Med 2011;154:752-7. [PubMed]
6. Horsley T, Hyde C, Santesso N, Parkes J, Milne R, Stewart R. Teaching critical appraisal skills in healthcare settings. Cochrane Database Syst Rev 2011;11: CD001270. [PubMed]
7. McCormack L, Sheridan S, Lewis M, Boudewyns V, Melvin CL, Kistler C. Communication and dissemination strategies to facilitate the use of health-related evidence. Evidence Reports/Technology Assessments No 213. US Agency for Healthcare Research and Quality, 2013. [PMC free article][PubMed]
8. Timmermans S, Berg M. The gold standard: the challenge of evidence-based medicine and standardization in health care. Temple University Press, 2003.
9. British Thoracic Society. Guidelines for management of asthma in adults: I. Chronic persistent asthma. BMJ 1990;301:651-3. [PMC free article][PubMed]
10. Majeed A, Ferguson J, Field J. Prescribing of beta-2 agonists and inhaled steroids in England: trends between 1992 and 1998, and association with material deprivation, chronic illness and asthma mortality rates. J Pub Health Med 1999;21:395-400. [PubMed]
11. Kelly MP, Capewell S. Relative contributions of changes in risk factors and treatment to the reduction in coronary heart disease mortality. Health Development Agency, 2004.
12. Lau BD, Haut ER. Practices to prevent venous thromboembolism: a brief review. BMJ Qual Safety 2014;23:187-95. [PMC free article][PubMed]
13. Gray JAM. NHS Atlas of variation in healthcare. 2011. www.rightcare.nhs.uk/index.php/atlas/atlas-of-variation-2011/.
14. Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124. [PMC free article][PubMed]
15. Greenhalgh T. Why do we always end up here? Evidence-based medicine’s conceptual cul-de-sacs and some off-road alternative routes. J Primary Health Care 2012;4:92-7. [PubMed]
16. Basson R, McInnes R, Smith MD, Hodgson G, Koppiker N. Efficacy and safety of sildenafil citrate in women with sexual dysfunction associated with female sexual arousal disorder. J Women’s Health Gender Based Med 2002;11:367-77. [PubMed]
17. Leyden J, Dunlap F, Miller B, Winters P, Lebwohl M, Hecker D. Finasteride in the treatment of men with frontal male pattern hair loss. J Am Acad Dermatol 1999;40:930-7. [PubMed]
18. Bone HG, Greenspan SL, McKeever C, Bell N, Davidson M, Downs RW.Alendronate and estrogen effects in postmenopausal women with low bone mineral density 1. J Clin Endocrinol Metab 2000;85:720-6. [PubMed]
19. Cohen D. FDA official: “clinical trial system is broken.” BMJ 2013;347:f6980. [PubMed]
20. Every‐Palmer S, Howick J. How evidence‐based medicine is failing due to biased trials and selective publication. J Eval Clin Pract 2014. May 12. [Epub ahead of print.]
21. Heres S, Davis J, Maino K, et al. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. Am J Psychiatry 2006;163:185-94. [PubMed]
22. Turner EH, Matthews AM, Linardatos E, Tell R, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60. [PubMed]
23. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. [PMC free article][PubMed]
24. Le Couteur DG, Doust J, Creasey H, Brayne C.. Political drive to screen for pre-dementia: not evidence based and ignores the harms of diagnosis. BMJ 2013;347:f5125. [PubMed]
25. Krogsbøll LT, Jørgensen KJ, Grønhøj Larsen C, Larsen C, Gøtzsche PC.. General health checks in adults for reducing morbidity and mortality from disease. Cochrane Database Syst Rev 2012;10:CD009009. [PubMed]
26. Moynihan R, Doust J, Henry D. Preventing overdiagnosis: how to stop harming the healthy. BMJ 2012;344:e3502. [PubMed]
27. Allen D, Harkins K. Too much guidance? Lancet 2005;365:1768. [PubMed]
28. Palella FJ Jr, Delaney KM, Moorman AC, Loveless MO, Fuhrer J, Satten GA, et al. Declining morbidity and mortality among patients with advanced human immunodeficiency virus infection. HIV Outpatient Study Investigators. N Engl J Med 1998;338:853-60. [PubMed]
29. NICE. Dysepsia: managing dyspepsia for adults in primary care (GC17). 2004. http://guidance.nice.org.uk/CG17/Guidance/pdf/English.
30. Gatta L, Vakil NB, Scarpignato C, Fiorini G, Castelli V, Vaira D. Sequential therapy for eradication of Helicobacter pylori infection in adults: still far from the ideal treatment. a systematic review and meta-analysis. Gastroenterology 2013;144:S332.
31. Hart JE, Jeon CY, Ivers LC, Behlforouz HL, Caldas A, Drobac PC, et al. Effect of directly observed therapy for highly active antiretroviral therapy on virologic, immunologic, and adherence outcomes: a meta-analysis and systematic review. JAIDS 2010;54:167-79. [PMC free article][PubMed]
32. Swinglehurst D, Greenhalgh T, Roberts C. Computer templates in chronic disease management: ethnographic case study in general practice. BMJ Open 2012;2:e001754. [PMC free article][PubMed]
33. Glasziou P, Moynihan R, Richards T, Godlee F. Too much medicine; too little care. BMJ 2013;347:f4247. [PubMed]
34. Harrison S, Checkland K. Evidence-based practice in UK health policy. Routledge, 2009.
35. Doran T, Fullwood C, Kontopantelis E, Reeves D. Effect of financial incentives on inequalities in the delivery of primary clinical care in England: analysis of clinical activity indicators for the quality and outcomes framework. Lancet 2008;372:728-36. [PubMed]
36. Gillam S. Expert commentary. Pay for performance in UK general practice—the ambiguous impact of the Quality and Outcomes Framework. 2011. www.qualitymeasures.ahrq.gov/expert/expert-commentary.aspx?id=25658&search=Primary±Health±Care±.
37. Huntley AL, Johnson R, Purdy S, Valderas JM, Salisbury C. Measures of multimorbidity and morbidity burden for use in primary care and community settings: a systematic review and guide. Ann Fam Med 2012;10:134-41. [PMC free article][PubMed]
38. Duerden M, Avery T, Payne R. Polypharmacy and medicines optimization: making it safe and sound. King’s Fund, 2013.
39. Montgomery K. How doctors think: clinical judgement and the practice of medicine. Oxford University Press, 2006.
40. McNutt R, Handler NM. How clinical guidelines fail both doctors and patients. Scientific American Blog Network, 22 November 2013. http://blogs.scientificamerican.com/guest-blog/2013/11/22/how-clinical-guidelines-can-fail-both-doctors-and-patients/.
41. Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. Helping doctors and patients make sense of health statistics. Psychol Sci Public Interest 2007;8:53-96. [PubMed]
42. Dreyfus HL, Dreyfus SE, Zadeh LA. Mind over machine: the power of human intuition and expertise in the era of the computer. IEEE Expert 1987;2:110-11.
43. Llewelyn H, Ang HA, Lewis D, Al-Abdullah A. Oxford Handbook of Clinical Diagnosis. Oxford University Press, 2014.
44. Frank AW. Just listening: narrative and deep illness. Families Systems Health 1998;16:197.
45. Schei E. Doctoring as leadership: the power to heal. Perspect Biol Med 2006;49:393-406. [PubMed]
46. Hinder S, Greenhalgh T. “This does my head in.” Ethnographic study of self-management by people with diabetes. BMC Health Serv Res 2012;12:83. [PMC free article][PubMed]
47. Rosling H, Zhang Z. Health advocacy with Gapminder animated statistics. J Epidemiol Global Health 2011;1:11-4. [PubMed]
48. Richards T, Montori VM, Godlee F, Lapsley P, Paul D. Let the patient revolution begin. BMJ 2013;346:f2614. [PubMed]
49. Gierisch JM, Myers ER, Schmit KM, McCrory DC, Coeytaux RR, Crowley MJ, et al. Prioritization of patient-centered comparative effectiveness research for osteoarthritis. Ann Intern Med 2014. May 13. [Epub ahead of print.] [PubMed]
50. National Institute for Health and Clinical Excellence. Psoriasis: the assessment and management of psoriasis. CG153. NICE, 2012.
51. Green ML. Evidence‐based medicine training in graduate medical education: past, present and future. J Eval Clin Pract 2000;6:121-38. [PubMed]
52. Spiegelhalter D, Pearson M, Short I. Visualizing uncertainty about the future. Science 2011;333:1393-400. [PubMed]
53. Gabbay J, May Al. Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care. BMJ 2004;329:1013. [PMC free article][PubMed]
54. Supporting adoption of evidence into practice. MeReC Bull 2011;22:2-7. www.npc.nhs.uk/merec/therap/other/merec_bulletin_vol22_no2.php.
55. Lavis JN, Davies HT, Gruen RL, Walshe K, Farquhar CM. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy 2006;1:21-33. [PMC free article][PubMed]
56. Noble D, Mathur R, Dent T, Meads C, Greenhalgh T. Risk models and scores for type 2 diabetes: systematic review. BMJ 2011;343:d7163. [PMC free article][PubMed]
57. Garcia-Retamero R, Galesic M. Who profits from visual aids? Overcoming challenges in people’s understanding of risks. Soc Sci Med 2010;70:1019-25. [PubMed]
58. Elwyn G, Lloyd A, Joseph-Williams N, Cording E, Thomson R, Durand MA, et al. Option grids: shared decision making made easier. Patient Educ Counsel 2013;90:207-12. [PubMed]
59. Volk R, Llewellyn-Thomas H, Stacey D, Elwyn, G. The International Patient Decision Aid Standards (IPDAS) Collaboration quality dimensions: theoretical rationales, current evidence, and emerging issues. BMC Med Inform Decision Making 2013:S1-14.
60. Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. BMJ 1986;292:746. [PMC free article][PubMed]
61. Goff DC, Lloyd-Jones DM, Bennett G, O’Donnell CJ, Coady S, Robinson J, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk. J Am Coll Cardiol 2013. Nov 12. [Epub ahead of print.]
62. Montori VM, Brito JP, Ting HH. Patient-centered and practical application of new high cholesterol guidelines to prevent cardiovascular disease. JAMA 2014;311:465-6. [PubMed]
63. Lenzer J. Why we can’t trust clinical guidelines. BMJ 2013;346:f3830. [PubMed]
64. Mol A. The logic of care: health and the problem of patient choice. Routledge, 2008.
65. Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci 2010;5:56. [PMC free article][PubMed]
66. West CP, Dyrbye LN, Rabatin JT, Call TG, Davidson JH, Multari A, et al. Intervention to promote physician well-being, job satisfaction, and professionalism: a randomized clinical trial. JAMA Intern Med 2014;174:527-33. [PubMed]
67. Eysenbach G, Powell J, Englesakis M, Rizo C, Stern A. Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. BMJ 2004;328:1166. [PMC free article][PubMed]
68. Chanchlani N, Godlee F. Educating tomorrow’s doctors. BMJ 2012;344:e3689. [PubMed]
69. Cooke M, Irby DM, O’Brien BC. Educating physicians: a call for reform of medical school and residency. Wiley, 2010.