Louise Vinter, Foreign & Commonwealth Office, and David Knox, British Council

Executive summary

The Foreign & Commonwealth Office (FCO) and the British Council have designed a pilot framework for planning and evaluating public diplomacy activity. This chapter sets out the rationale for the framework and how it is being tested.

The last ten years have seen a significant shift towards evidence-based policymaking in the UK. However, in the area of foreign policy, and public diplomacy in particular, there has been much debate over the extent to which measurement and evaluation techniques can, and should, be applied. In order to test some of the assumptions involved, the FCO and the British Council are jointly piloting, over a two-year period from April 2007, an evaluation framework designed to measure the impact of their public diplomacy activities. It is too early to draw conclusions, but this chapter sets out the rationale and techniques underpinning the experiment, in the hope that others working in the field can debate and draw on these.

The importance of measurement

The move towards evidence-based policy in the UK began in the late 1990s.[1] The 1999 White Paper Modernising government set out the requirement in the following terms: ‘This Government expects more of policy makers. More new ideas, more willingness to question inherited ways of doing things, better use of evidence and research in policy making and better focus on policies that will deliver long term goals.’[2]

This principle was reflected in the independent review of public diplomacy commissioned by the FCO from Lord Carter of Coles in 2005. Lord Carter remarked that ‘A better central strategy and an improved system for collective monitoring and evaluation should enable an informed assessment of the impact and value for money of public diplomacy efforts as a whole.’[3] Lord Carter reasoned that improved measurement would not only provide a means of accountability but also enable policy-makers to develop strategy, and deploy resources, more effectively. At the operational level, it would give managers targeted feedback to enable them to improve performance and focus efforts on those kinds of intervention that have been shown to be most effective.

Where we started from

As a result of the Carter Review, a Public Diplomacy Board was established, comprising representatives of the FCO, the British Council and the BBC World Service, together with independent members from the private sector. One of its earliest decisions was to investigate the feasibility of a shared measurement system for public diplomacy, beginning with a review of systems already in place.

The FCO

The review team found that while the FCO did conduct regular narrative reporting against objectives, there was no accurate measure either of the effectiveness of public diplomacy activity in supporting FCO policy objectives or of the resources spent on it. A variety of evaluation techniques (such as opinion polling and media analysis) were being used, particularly in larger overseas missions and for bigger campaigns. But there was no systematic approach.

The British Council

The British Council had a more developed system in place. It operates a corporate performance scorecard, assessing performance across a range of factors including project impact, customer and stakeholder satisfaction, reputation, financial and management results, and the perceptions of staff. This focuses mainly on outputs, but, given that much of the Council’s work requires the development of long-term relationships, the Council also uses evaluative research with leaders and influencers to try to measure longer-term outcomes (see Figure 12.1).

Figure 12.1: The British Council’s corporate performance scorecard

The difficulty of measurement

From the start, we were conscious of the obstacles. As Tim Banfield, Director of Value for Money Studies at the National Audit Office, has observed in relation to the British Council’s work: ‘Public diplomacy is about building relationships between diverse nations and cultures, and these are constantly influenced by many external factors. And because the full effect of the Council’s activities may only become evident after long periods, its changing impact is very difficult to measure year-on-year.’

There are three inherent difficulties in measuring public diplomacy: its frequently long-term ambition; the challenge of measuring concepts that may be intangible; and the problem of attributing observable changes to one’s own activities.

Professor Nick Cull has commented, in reference to the timescales involved in cultural diplomacy (and the same can be said of much public diplomacy): ‘Attempts to evaluate cultural diplomacy can seem like a forester running out every morning to see how far his trees have grown overnight.’[4] Just as it would make little sense to require the forester to measure his trees daily, so it would be impractical to ask missions to quantify progress against such long-term policy goals as ‘promoting a low carbon, high growth global economy’, or ‘countering terrorism, weapons proliferation and their causes’.[5]

The intangible nature of some public diplomacy objectives, such as increased ‘trust’ or improved ‘relationships’, adds further complication. To expand on the observation by Nick Cull, it makes attempts at evaluation seem like a forester going out to measure how far his trees have grown overnight without a ruler.

The question of attribution is the most difficult challenge of all. In a world where multiple organisations and influences are acting upon the same complex policy issues, how do we identify changes that can be attributed to our own actions, and, more specifically, to our public diplomacy activities? To add to this complexity, can or should we distinguish the impact of public diplomacy from that of other diplomatic work? In addition, there may be public diplomacy activity deliberately conducted at arm’s length from government and for which public attribution would not be welcome.

The new framework

With these challenges firmly in mind, the FCO and the British Council set about the task of creating and piloting a new evaluation framework.

The first, important, prerequisite was the development of joint FCO and British Council public diplomacy strategies in a number of agreed pilot countries, with the two organisations working together to reach jointly agreed outcomes through their engagement with non-governmental audiences. These strategies were developed using a logical framework which traces a course from the UK government’s policy goals, through long-term outcomes, intermediate outcomes, outputs and activities, and back to inputs. This logical framework provides a common way of thinking about strategy, and the use of a common language helps the two organisations to develop a shared understanding of what they are trying to measure and how (see Figure 12.2).

Figure 12.2: The logical framework for country public diplomacy strategies

Crucially, this framework also brings consistency to the way that public diplomacy activities are monitored and reported across the two organisations. At the level of input, resources are measured in terms of staff time and direct project spend. At the level of activity or outputs, the focus is on a systematic approach to monitoring media coverage, to the collection of feedback from participants in public diplomacy activities and to follow-up evaluation after completion of each such activity. The emphasis is at all times on evidence-based evaluation rather than narrative reporting.

It is at the level of intermediate outcome that one begins to home in on the measurement of ‘impact’, using a combination of three evaluation tools or ‘trackers’:

• a media tracker which seeks to identify changes in the nature and tone of coverage of targeted issues, and, where possible, the reasons for these changes;
  
• an influencer tracker to generate information on opinion change among those individuals considered key ‘influencers’ on policy issues related to the intermediate outcomes. This involves systematic mapping of influencers and semi-structured interviews, repeating the process year on year in order to track changes in opinion; and
  
• a concrete changes tracker for recording objectively verifiable changes in the environment that are related to the intermediate outcomes, whether positive or negative.

Figure 12.3 summarises how this evaluation framework fits into the logical framework for country public diplomacy strategies, illustrating how important it is for the strategic planning to have been done properly if the evaluation tools are to produce meaningful results.

Figure 12.3: The relationship between the evaluation framework and the logical framework for planning public diplomacy activity

The evaluation framework tackles the challenge of long-term ambition through its treatment of public diplomacy as a journey from input to policy goal, indicating the various staging posts along the way. The articulation of the links between each stage and the next is critical to the validity of the evaluation process. Intermediate outcomes, the central focus for the measurement of impact, are staging posts in this sense and are defined as the medium-term changes (0–5 years) which the outputs from a programme of activity are expected to help deliver. They allow impact to be evaluated from a shorter-term perspective while still enabling the overall ‘direction of travel’ to be assessed in relation to longer-term outcomes.

The framework addresses the intangibility of certain measures by requiring strategies to be expressed in terms of outputs and intermediate outcomes that are achievable changes. In the case of the influencer tracker, the use of semi-structured interviews provides a means to explore complex issues with more nuance and shade.

The problem of attribution is managed by the recognition that public diplomacy activities are never carried out in isolation – and that what matters is in fact contribution rather than attribution. In analysing the evaluation data, one may need to interpret what are sometimes fairly weak signals. As Colin Wilding acknowledges in a paper on the subject, ‘it may well be possible to demonstrate that public diplomacy activities have made a positive contribution even if the magnitude of the effect cannot be quantified precisely’.[6]

Overall, the framework seeks to ‘triangulate’ evidence using the three tracking tools so that, if there is a policy change, we will be able to capture this through recording concrete outcomes, see it reflected in the media, and test the significance of the change and the part played by public diplomacy through the influencer tracker interviews. The tools also try to identify the context of any observed change, and the other influences involved.

Early results

The pilot framework is still in the early stages of implementation, so it would be premature to start drawing conclusions about the extent to which it will succeed in enhancing our ability to measure the impact of public 169 diplomacy activity; accordingly, we do not in this chapter attempt to draw such conclusions.

The use of the new measurement framework has, however, helped to identify some links between activities, outputs and progress towards the outcomes we are seeking to achieve.

For example, in the area of evaluating media coverage, we have been using standard measures of reach, tone and prominence across all pilot countries, and have been able to pick up much more systematically the relationship between media campaigns, as part of individual strategies, and media coverage. This is then considered alongside more detailed content analysis from the media tracker, in order to assess the effectiveness of that coverage in influencing the overall media debate on an issue.

The structured sequencing of evaluation has also helped us to develop an evidence base linking outputs to outcomes. If we take the example of young people who participated in a climate change event, all reported – at the ‘output’ level – increased awareness and understanding of climate change. In follow-up interviews to explore outcomes, they reported behavioural changes in relation to their own carbon footprint, and, more importantly, all said they were engaging other people within their community to take action to tackle climate change. Another initiative has led to the development of new teaching and learning materials exploring the science of climate change, which have subsequently been used in schools, reaching thousands of children. In another example, activities engaging local government officials, again on the issue of climate change, can be seen to have led directly to the development by local governments of plans for making a transition towards a low-carbon economy.

Early experience has also shown some practical difficulties in implementing the evaluation framework. In particular, the influencer tracker is a challenging tool requiring correct identification or mapping of influencers, interviews with often quite senior individuals, and analysis of qualitative and quantitative data. Even though the research itself is outsourced, there are still implications for our organisations in terms of time and resources to manage the process.

Conclusion

This chapter has tried to set out, in brief, the rationale underpinning the joint work of the FCO and the British Council to construct and pilot a new planning and evaluation framework for public diplomacy activity. A key principle has been that ‘the systematic approach to planning can be expected to deliver benefits even if it proves to be impossible to establish strong causal links all the way along the chain from inputs to longer-term outcomes’.[7]

The aim is not a central data-gathering exercise but a framework that enables practitioners to understand the impact of their actions and provides an evidence base for decisions about strategy and activity. Given its expense, the framework is not intended as a tool to be employed comprehensively across the network. But we hope it will provide sufficient insight into what works and what doesn’t to allow policy-makers to make more informed decisions in the planning of their public diplomacy activity.

Notes

1. William Solesbury, Evidence based policy: whence it came and where it’s going (ESRC UK Centre for Evidence Based Policy and Practice, Oct. 2001).
   
2. Cabinet Office, Modernising government, Cm 4310 (London: Stationery Office, 1999), ch. 2, para. 6. Available at http://archive.cabinetoffice.gov.uk/moderngov/whtpaper/index.htm
   
3. Lord Carter of Coles, Public Diplomacy Review, Dec. 2005.
   
4. Nicholas J. Cull, Public diplomacy: lessons from the past (Los Angeles: University of Southern California, April 2007).
   
5. These are two of the FCO’s four policy goals identified in FCO’s Strategic Framework, which is in place from 1 April 2008.
   
6. Colin Wilding, ‘Measuring the effectiveness of public diplomacy: the UK approach’, paper presented to the Annual Conference of International Radio Broadcasters, Nov. 2007.
   
7. Wilding, ‘Measuring the effectiveness of public diplomacy’.