Grading Evidence-Based Guidelines—What Are the Issues?

What is the USPSTF Proposing? 

Table 1 summarizes the sequence of grading steps followed by USPSTF. For each overarching question related to a preventive service, relevant studies related to screening, treatment benefits, and harms have to be reviewed. The quality for each body of evidence is assessed according to criteria detailed in Table 2. The magnitude of the net overall benefit is then estimated across all relevant bodies of evidence and the certainty surrounding this estimate is determined by again applying the criteria in Table 2. Based on the estimated magnitude of the net overall benefit and the certainty surrounding this estimate, the USPSTF assigns a letter grade from “A” to “D” to the preventive service. The letter signifies the strength of the recommendation about whether the service should be provided. An example is the recent recommendation by the Task Force on carotid artery stenosis screening.6 The Task Force rated as “moderate” the certainty about the net benefits from screening followed by treatment. It judged the magnitude of net benefit as “zero or negative,” and issued a “D” grade recommendation that the screening should not be provided in asymptomatic people.6

Table 1. Grades for Quality of Evidence and Strength of Recommendations by USPSTF, GRADE, and KDIGO

	Steps for Grading or Assessment	USPSTF	GRADE	KDIGO
	Quality of individual study	Good Fair Poor	Not specified	Good Fair Poor
	Quality of evidence for each key question in analytic framework (USPSTF) or for each outcome (GRADE and KDIGO)⁎	Convincing Adequate Inadequate	High Moderate Low Very Low	High Moderate Low Very Low
	Overall certainty of net benefit (USPSTF) or overall quality of evidence across all important outcomes (GRADE and KDIGO)⁎	High Moderate Low	High Moderate Low Very Low	High Moderate Low Very Low
	Magnitude of net benefit	Substantial Moderate Small Zero/Negative	Net benefit Important trade-offs Uncertain trade-offs No net benefit	Net benefit Important trade-offs Uncertain trade-offs No net benefit
	Strength and meaning of recommendation	A = Should be provided for eligible patients B = Should be provided for eligible patients C = Should not be offered routinely D = Should not be provided I = Insufficient evidence	Strong = “Do it!” or “Do not do it!” Weak = “Probably do it!” or “Probably do not do it!”	A = “Should be done”† B = “Should be considered”† C = “Is suggested”†

Abbreviations: GRADE, Grades of Recommendation Assessment Development and Evaluation; KDIGO, Kidney Disease Improving Global Outcomes; USPSTF, US Preventive Services Task Force.

⁎ Criteria used for assessment are listed in Table 2. † These strengths can also be used for negative recommendations, but the wording for these has not yet been standardized in KDIGO.

Table 2. Criteria for Assessing the Quality of Evidence

		USPSTF	GRADE and KDIGO
	Matching question of interest with research design	Appropriateness of the study design of the reviewed studies.	Appropriateness of the study design of the reviewed studies.
	Internal validity	Methodological quality.	Methodological quality.
	External validity	Generalizability to the general primary care population and situation in the United States.	Directness of the evidence, ie, similarity of populations, interventions, outcome measures, to those of interest.
	Precision of evidence	Quantity and size of studies.	Precision and lack of sparseness of results.
	Consistency	Consistency of results across studies.	Consistency of results across studies.
	Additional factors	Dose-response effect, fit within a biological model.	Dose-response gradient; plausible residual confounders would have reduced observed effect; strength of association (for observational studies); lack of reporting bias.

Note: These criteria are used in 2 grading steps: 1) to assess the quality of evidence for a key question (USPSTF) or the quality of a body of evidence for an outcome (GRADE and KDIGO); 2) to grade the overall certainty for net benefit across key questions (USPSTF) or the quality of evidence across the bodies of evidence for all important outcomes (GRADE and KDIGO). Abbreviations: GRADE, Grades of Recommendation Assessment Development and Evaluation; KDIGO, Kidney Disease Improving Global Outcomes; USPSTF, US Preventive Services Task Force.

How Does the USPSTF Approach Compare to GRADE and KDIGO? 

As shown in Table 1, Table 2, USPSTF, GRADE, and KDIGO use broadly similar but distinctive approaches toward grading the quality of evidence and the strength of recommendations. The sequential steps used by all 3 groups highlight that guideline development follows a complex process. As detailed in the USPSTF approach, the strength of a recommendation relates to the size of the overall net effect as well as the certainty surrounding the estimate, which is based on the quality of the overall supporting evidence. However, no system has a formulaic approach to sum up all assessments. Furthermore, values and preferences are applied by the guideline development committee in each step of appraisal and generate a degree of arbitrariness. Implicit or explicit consideration of costs adds an additional layer of complexity.3 The focus for the USPSTF recommendations is the United States, while the context for KDIGO recommendations is global, and therefore values and judgments will inherently vary more widely.

These issues become even more complex when interpreting inconclusive evidence. Consider the question of whether dialysis patients with hepatitis C infection should be treated with interferon.7 There is moderate-quality evidence that interferon treatment results in a sustained viral response (SVR) in about 40% of the patients.8 There is, however, uncertainty whether achieving SVR reduces mortality or improves quality of life in patients on dialysis or after kidney transplantation. Furthermore, the treatment has frequent and sometimes severe adverse events. The treatment duration prolongs the waiting time and may close the window of opportunity to transplantation. Finally, the treatment is costly.

In the above scenario, if a guideline workgroup values SVR highly because it believes that SVR is a good surrogate for better survival and quality of life and deemphasizes adverse events because they are usually self limited or manageable, it may issue a recommendation for the treatment. If, however, the workgroup judges the benefits of SVR to be uncertain and is more concerned about proven harms associated with interferon therapy and also its direct and indirect costs, it may issue a weak recommendation against the use of the treatment. Alternatively, a recommendation could advise using the treatment only in selected patients where the benefit-harm ratio is most favorable (for example, transplant candidates with longer life expectancy), or the workgroup could refrain from making any recommendation. Therefore, despite examining identical evidence, different workgroups could reasonably arrive at discordant recommendations or at recommendations of different strengths.

This example illustrates how when interpreting inconclusive evidence, workgroup dynamics9 and members' philosophies and attitudes towards uncertainty10 can determine whether and how a recommendation is issued. At one end of the spectrum are workgroup members who feel that even if the evidence is not conclusive, it is still useful to suggest what makes sense or is currently acceptable practice. After all, not many practitioners will have the time and expertise to systematically review and appraise the primary literature to inform their own opinions. At the other end of the spectrum are members who hesitate to issue a recommendation that is predominantly based on expert opinion, no matter how its strength is rated, given the possibility of being proven wrong by future research. The direction provided by the guideline-issuing body to its workgroup will further impact on the final guideline product. For example, the KDIGO Board allows its workgroups to issue recommendations based predominantly on expert judgment in the setting of insufficient evidence.

Beyond the philosophical debate in guideline development, there is a tension between complexity and simplicity in grading. Guideline developers must follow a complex process, incorporate judgments, and transparently record decisions. But users want succinct guidelines with simple and intuitive grades for recommendations. With the proliferation of guideline initiatives, different grading systems have been developed.11, 12, 13 The coexistence of different systems presents an apparent paradox in evidence-based practice, given the overall goal of improving patient care by reducing unjustified variability. While common sense suggests that there is benefit to having one convention for grading, a single system that fits all purposes and transparently summarizes a multidimensional process in a simple, unequivocal, and intuitively understood set of grades is likely to remain an elusive goal.

What Should Developers, Users, and Researchers of Guidelines Do? 

Given the imperfection of any grading system, one might ask whether guideline recommendations should be graded at all. The answer depends on one's assessment of whether guideline users can and prefer to independently synthesize and appraise the evidence to inform their decisions. Inasmuch as there is an audience for guidelines, it can be assumed that their usefulness is enhanced by being explicit about their strength. While there is no direct evidence proving the benefit of grading versus not grading recommendations in terms of clinical outcomes, an unpublished, crossover, randomized, controlled trial by UpToDate demonstrated a strong preference by its users for graded recommendations (P. Bonis, personal communication, November, 2007).

Entities commissioning guidelines should be explicit in outlining the charge to the guideline workgroups, in particular their expectations regarding whether recommendations should be issued when evidence is not conclusive. Guideline developers should acknowledge that grading is a process that includes some degree of subjectivity and that following a structured approach is useful to guide the deliberations. The guideline workgroup needs to be explicit about its values and preferences, especially in areas of lower-quality evidence. Furthermore, the workgroup needs to define what each grade means and which recommendations are or are not reasonable candidates for translation into performance measures. If the committee issues recommendations in areas of inconclusive evidence that are predominantly based on expert opinion, these should be clearly differentiated from guidelines supported by higher-quality evidence.

Guideline users need to develop an understanding for the complexity of the process and become literate about the issues that impact the strength of a recommendation.14 It is vital that the implications for recommendations of different strengths are understood, along with the degree to which they are based on a workgroup's judgments. The more sensitive a recommendation is to values and preferences, the greater is the need for the practitioner to incorporate the values and preferences of the patient when deciding whether to follow a recommendation.

Researchers of guideline methods should empirically evaluate the utility of a grading system. Surveying guideline users will generate a better understanding of their preferences and needs. It can also provide information on how grading or formulating the strength of a recommendation affects guideline uptake and implementation in clinical practice.