Guidance for Evidence-Informed Policies about Health Systems: Rationale for and Challenges of Guidance Development

Summary

article has not abstract

This is one paper in a three-part series that sets out how evidence should be translated into guidance to inform policies on health systems and improve the delivery of clinical and public health interventions.

Introduction

Present trends suggest that many of the poorest countries in the world, including many in sub-Saharan Africa, will not meet the health-related Millennium Development Goals [1] (MDGs), especially MDG 4 (reducing under-five mortality) and MDG 5 (reducing maternal mortality) [2]. Even in those countries that are on track to meet health MDGs, striking inequities exist among countries and among socioeconomic groups within them [3], despite effective and cost-effective interventions being available to improve population health, including that of vulnerable groups [4]. Such interventions are delivered through health systems, which consist of “all organisations, people and actions whose primary intent is to promote, restore or maintain health” [5], but, in many settings, interactions between weakened health systems and the sometimes conflicting demands of single-disease intervention programmes are hindering the uptake and implementation of life-saving interventions [6]–[8]. A growing number of governments, international institutions, and funding agencies have therefore recognised the urgent need to coordinate and harmonise investments in health systems strengthening in low- and middle-income countries (LMICs) to provide universal social protection and effective coverage of essential health interventions [9].

Investments in health systems aim to “enhance [their] performance…for meeting the needs of patients and populations in an equitable and efficient manner” [10] while reducing the risk of impoverishment due to the costs of care [11]. However, although a number of broad principles have been proposed [12], there is no wide agreement on the operational definition of health systems strengthening [13], and it remains unclear how health systems can best be strengthened. Because the evidence base addressing this issue is patchy [14], health systems research has recently been identified as a priority [15] and its definition and scope have been outlined [16]. Indeed, the need for greater capacity to produce evidence to inform health systems strengthening was one of the drivers that led to the first global symposium on health systems research (Montreux, Switzerland, November 2010) [17], at which some of the issues developed in this article were presented and discussed.

Importantly, to be useful to policy makers, research evidence needs to be retrieved, its quality appraised, and the recommended options properly framed in the form of guidance. In an analogy with clinical practice guidelines (“systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances” [18]), we define health systems guidance as systematically developed statements produced at global or national levels to assist decisions about appropriate options for addressing a health systems challenge in a range of settings and to assist with the implementation of these options and their monitoring and evaluation (Box 1). We use the term “guidance” rather than “guidelines”, as health systems and the evidence on health systems are highly context sensitive. Health systems guidance statements refer to policy options that are accompanied by assessments of the quality of evidence supporting them and the potential for unintended harms, and by discussions of implementation and contextual issues.

The need for evidence-informed guidance on policies that impact health systems performance is widely accepted [19] and is one of the six priorities of the director-general of the World Health Organization (WHO) [5]. Moreover, a World Health Assembly resolution [20] recently urged member states to use evidence-based approaches to assess “country's health and health systems challenges” and to develop “evidence-based responses to evolving challenges and opportunities, and to involve all relevant stakeholders”. However, although well-established methods exist to develop clinical guidelines [21], there is little experience in developing health systems guidance and the process poses conceptual and methodological challenges related to the different types of evidence to be considered, the complexity of health systems and the pre-eminence of contextual issues. The current experience is mainly related to the development of different by-products of research syntheses and decision aids [22] targeted towards policy makers, rather than systematically and transparently developed guidance.

This paper, which is the first in a three-part series on health systems guidance [23],[24], aims to:

• Assess to what extent the need for health systems guidance is part of national policies and plans and how guidance is currently formulated by analyzing strategic health sector documents from LMICs;

• Describe the methodological challenges in outlining the approaches to produce health systems guidance and to suggest ways to address these challenges.

The second article in this series explores the challenge of linking guidance development and policy development at global and national levels and examines the range of factors that can influence policy development [23], and the third article explores the challenge of assessing how much confidence to place in evidence on health systems interventions [24].

The Need for Health Systems Guidance in LMICs

To assess the need for guidance and how it is formulated, we scrutinised the use of guidance-related terms in national health policy and strategic documents from LMICs (Box 2). We found that the terms “guidance” or “guidelines” frequently appeared in strategic documents but were more often related to clinical matters than to health systems (Box 3).

Challenges in Outlining the Approaches to Produce Health Systems Guidance

To improve how WHO responds to requests for guidance on health systems and the quality of the guidance produced, WHO recently commissioned the production of a Handbook outlining approaches to develop health systems guidance [25]. A Task Force on Developing Health Systems Guidance (listed in the Acknowledgments section) was formed from WHO expert panels and from experts working in the field of health systems research at international and academic institutions and oversaw the drafting of the Handbook for Developing Health Systems Guidance: Supporting Informed Judgements for Health Systems Policies [25] (which was based on existing best practices for the development of clinical guidelines, on approaches and tools described in the literature, and on the expertise of the Task Force members) by reviewing and commenting on its content by email and through regular teleconferences and meetings in person. During the production of the Handbook, because evidence from research (for example, on the effectiveness and acceptability of health systems interventions) is not guidance and is insufficient for optimal decision-making [26], several conceptual and methodological challenges associated with the production of health systems guidance were identified. The writing group for this paper further considered these issues and produced this manuscript, which was finalised after several iterations of comments by the Task Force and external reviewers. Here, we discuss four specific challenges that were encountered during the production of the Handbook, namely:

• Research on effectiveness is typically articulated around health interventions, but policy decisions often relate to arrangements of the health system, services, and programmes and encompass multiple interventions packaged into a particular policy;

• Users and producers of guidance include a broad range of stakeholders who may not all be familiar with research methods;

• The production of guidance has to be timely in relation to the need and available capacity to implement it and consistent with countries' priorities;

• The quality of guidance needs to be appraised using transparent criteria; guidance then needs to be disseminated and promoted actively to facilitate its uptake.

Translating Research on Problems, Interventions, and Implementation into Decisions on Policies and Services

Many processes, frequently involving iterations from research evidence into policy formulation and from policy evaluation into research prioritisation, are needed to bridge the gaps between research, policy, and practice. The complexity of these processes demands a dynamic framework [27] that is comprehensive and incorporates current thinking about evidence, policy formulation, and health systems such as the one shown in Figure 1.

Research is articulated around research questions that can be addressed using particular methods and is driven by research opportunities (e.g., funding), researchers' interests, and feasibility. By contrast, policy decisions and the managerial arrangements needed to put them into practice have to be responsive to population needs and integrated into complex health systems that go beyond the mere aggregation of single interventions. For example, the research question of whether lay health workers are effective in delivering specific health care interventions will become fully relevant for policy only when the essential components of the intervention (e.g., training of lay health workers), related actions (e.g., adaptations needed in the distributions of tasks of cadres), implementation issues (e.g., the preferences of potential clients), and the implications across other health system building blocks (e.g., adaptations to the health information sub-system that may be needed to capture the tasks undertaken by such workers) are all considered.

Ideally, implementation and contextual issues should be considered when conducting the systematic reviews of effectiveness that are needed to translate research into health systems guidance or in other types of reviews addressing them [28],[29]. Although there are frameworks to analyse barriers to implementation [30] and the applicability of evidence [31], many systematic reviews fail to consider issues pertaining to applicability and equity, or the complexity of interventions in relation to their technical feasibility [32]. This is partially due to the lack of this type of information in the primary research that underlies systematic reviews. Where this evidence is missing or scarce, efforts to search, appraise, and synthesise additional evidence on implementation issues will have to be made. It will also be important to consider how to address inequities in coverage of health care [30], which are pervasive in many countries.

The production of guidance not only entails putting research findings into context, but also entails taking a health systems perspective. New trends in health systems thinking advocate the construction of conceptual pathways that look at the interventions from a systems perspective, making explicit how interventions may trigger reactions in related components of the health system that may produce unintended consequences [33],[34]. For example, increasing the salary of health workers in HIV/AIDS treatment programmes may result in a reduction of the health workforce addressing other conditions who do not receive the same salary benefits.

Finally, it is important to note that the study designs for research (and the methods used to synthesise research findings) that explore different health systems components vary. For example, research exploring the governance and leadership components of health systems is likely to involve qualitative methods, whereas research on health care delivery could utilise both qualitative and quantitative methods (see also the third paper in this series [24]). Qualitative synthesis methods have addressed the nature of problems such as patient adherence to treatment for tuberculosis [35]. Syntheses of mixed method research can address policy-oriented questions using framework synthesis, a method that has been applied to qualitative and mixed methods research addressing research management [36], public health [37],[38], and workforce management [39].

Producers and Users of Guidance

Although research evidence to inform guidance is generated through research synthesis (i.e., systematic reviews) and often uses complex statistical methods, guidance is typically produced through a deliberative process where evidence is interpreted and contextualised (Figure 1, Policy domain). During the deliberative process, which can include a wide range of stakeholders, potential users, and beneficiaries of guidance, knowledge is exchanged, filtered (stakeholders decide on the relevance of evidence), and amplified (stakeholders stress evidence consistent with their views). Colloquial (non-research) evidence (e.g., tacit or experiential knowledge) helps to interpret or contextualise research evidence and addresses issues for which research evidence is not available but that may play a role in the decisions (e.g., considerations about the political implications of decisions, or about potential vested interests of stakeholders) [40]. Making colloquial evidence explicit adds transparency to the guidance development process.

Typically, users and producers of guidance, who may include economists, managers, administrators, social scientists and other professional groups as well as elected politicians, come from very different research traditions or none at all and have very different approaches to research and decision-making [41]. Therefore, it is probably unrealistic to expect that they will all be familiar with issues such as assessing the quality or appropriateness of systematic reviews [42]. Furthermore, even in the ideal situation where policy makers are familiar with research methods, they may choose to ignore evidence that creates uncertainty, questions conventional knowledge, ignores local context [43], or that is not consistent with pre-determined ideas of preferred policy options.

Although evidence on the effects of health systems interventions is relatively scarce, there are renewed efforts to increase availability and accessibility of this evidence [22],[44],[45]. Such evidence should be presented in user-friendly formats (e.g., minimising technicalities such as complex statistical outputs) to increase its accessibility when producing guidance and to facilitate appropriate interpretation by those without a strong research background. Finally, the preferences of different decision makers for how evidence and guidance should be presented should be taken into account.

Timeliness in the Production of Guidance

Timeliness in the production of evidence and guidance is critical, but systematic reviews can take between one and two years to complete depending on the availability of new evidence, and guidance development can take a year or more [46]. The bottom of Figure 1 also shows the potentially long time frames for the policy decision processes and the evaluation of policies. Furthermore, guidance needs to be regularly updated in the light of new evidence or changes and modifications to existing, previously accepted evidence.

Without timely production of research synthesis and guidance, the gap between research, policy, and practice cannot be bridged, even if primary research is available. Alternative approaches and methods are therefore needed to reduce the lengthy time frames needed for producing systematic reviews and for translating the findings of reviews into guidance. Approaches that may be useful include rapid assessment methods [47], the use of text mining to speed the search for publications [48], and the adaptation of existing guidance that can be institutionally endorsed by the organisations producing or using it [47].

Appraisal, Dissemination, and Implementation of Guidance

The quality of guidance has to be assessed to ascertain the extent to which the guidance used state-of-the-art and validated methods during its development and is, therefore, balanced and reliable in relation to the evidence that informs it. The AGREE instrument (Appraisal of Guidelines Research and Evaluation) and its recent revision (AGREE II) is used to assess the quality of clinical guidelines but could be adapted to assess health systems guidance. The AGREE instrument, which consists of 23 items, aims to assess the quality of guidelines, to provide a methodological strategy for the development of guidelines, and to inform guideline developers on what and how information ought to be reported in guidelines [49]. Many of its items are applicable to both clinical guidelines and health systems guidance, but some need minor adaptation for the latter. For example, item 16 in the AGREE II instrument mentions the “different options for the management of the condition or health issue are clearly presented” whereas in health systems guidance this would need to be reworded to indicate “different options for addressing the health system topic are clearly presented”. In addition, there should be specific reference to the need for health systems guidance to refer to the contextual factors that would determine the extent to which research evidence is applicable in specific circumstances.

Passive dissemination of guidance will not ensure its uptake by potential users [50], particularly if it lacks relevance to local or national situations. There are several proven approaches for the effective dissemination of research evidence and the promotion of clinical guidelines (e.g., distribution of educational materials, educational meetings and outreach visits, involvement of local opinion leaders, audit and feedback) that have moderate effects on outcomes [51]. Some of these approaches may also be relevant to health systems guidance, although the challenges relating to dissemination and uptake for these two types of guidance are different. At a national level, a relatively small number of decision makers may need to be influenced to integrate guidance into national health policies and plans. Alternatively, sub-national and local decision makers may be in a better position to influence guidance implementation, particularly in more decentralised systems. “Policy briefs” and “policy dialogues” are examples of tools used to improve the contextualisation and utilisation of guidance [41] (discussed further in the second article in this series [23]). Although uptake of guidance depends on how it is presented and disseminated, additional factors may increase the utilisation of scientific evidence by policy-makers. These include early, informal interactions between researchers and policy makers, the relevance and timeliness of evidence, and the consistency between the evidence and the recommended options on the one hand, and the beliefs, values, interests, or political goals of policy makers on the other [52].

Conclusions

We found that high-level policy and strategic health sector documents frequently mention the need for the development of health systems guidance but suggest that there may still be limited awareness that health systems guidance can provide key inputs into policy-making and that the development of such guidance requires systematic and transparent approaches, inspired by the development of clinical guidelines. However, given the growing amount of evidence on health systems and the current initiatives to address methodological challenges to produce health systems guidance (e.g., the Handbook for Developing Health Systems Guidance [25]), the international health community at all levels is likely to be faced with increasing demands for guidance on health systems issues.

The approaches we identified for producing guidance for health systems typically mirror validated methods developed for clinical guidelines but with some important differences. Specifically, to develop health systems guidance that is relevant and useful to decision makers, it is necessary to acknowledge and to find ways of incorporating the complex interrelations of the system components, and the numerous contextual factors that may influence the effectiveness of interventions, particularly their effects on disadvantaged populations. These issues are discussed further in the second paper in this series [23]. The uptake of guidance by decision makers and the additional complexity of the decision-making process itself are also important.

Health systems guidance has the potential to improve decision-making and enable more efficient use of resources with consequent improvements in the health of populations. However, such guidance needs rigorous and transparent processes of production and evidence-based approaches to ensure its dissemination and uptake. These processes and approaches are still at a rudimentary stage of development. Importantly, the availability of health systems guidance should encourage better and more comprehensive health systems research. This, in turn, should foster more cross-disciplinary approaches to studying the dynamic interactions within complex health systems and thus help to develop new methods to effectively translate health systems evidence into usable policy guidance that is relevant to stakeholders at global and national levels, that takes into account social and ethical imperatives, and that recognises the complexity of health systems and political systems within which such guidance is introduced.

Supporting Information

Attachment 1

Attachment 2

Attachment 3

Attachment 4

Attachment 5