It is a familiar experience for Fyusion to discover that a client organisation has many well considered processes for planning, designing and delivering services but none for evaluating whether they have actually achieved their goals. Evaluation is often the forgotten component of program (and project) management. This can occur for a number of reasons:
With deadlines to deliver to, the focus of the project team is often on meeting those deadlines and getting a service up and running rather than on what happens once the service is in place;
Defining good measures of impact is seen as difficult, as is gathering the required data;
Many people have had previous experience of being required to report against poorly designed KPIs, so are sceptical about the utility of measuring outcomes.
However, there is a new focus on evaluation within government which means that evaluation will no longer be optional. At Commonwealth level, the Department of Finance is implementing an extensive performance framework that has a strong focus on assessing impact and outcomes. In January this year, the NSW State Government published its Program Evaluation Guidelines to support agencies to evaluate NSW Government funded programs.
So, what can you do if you have never been involved in evaluation before? Both the Commonwealth and NSW Government initiatives are being delivered with a range of support materials to assist staff who have limited experience of evaluation. However, there are a couple of simple tips that can help when you are approaching evaluation:
The single most effective step you can take is to include evaluation planning in the early stages of planning for the program as a whole.
Ensure that you define the overall goals of the program clearly and map out the outcomes you want to achieve.
Consider the timeline for the program and think about the short term changes that you hope to see as against the longer term outcomes.
Work out how you can collect data to measure both short and longer term impacts and put these measures in place from the outset. Wherever possible, leverage from information that is already being collected.
Make sure that the measures you choose are practical and will help you to identify critical issues and make decisions about what needs to be adjusted if the program is not delivering on its objectives.
Remember that you don’t have to have a lot of measures in place if you choose the right ones. Some organisations are now finding that a single measure of customer service is all they need: they simply ask customers if they found it easy to interact with the organisation (for more on this read our ‘Simply Measuring Success!’ article below).
If you have not done this early planning, then it is never too late to start! While you might have lost the opportunity to develop customised impact measures, existing information sources can always be leveraged. You might find that you need to put together two or three existing measures to build a good picture of the program’s impact in a particular area. If this is the case, combining statistical data with more qualitative feedback can be very powerful.
Finally, the effort of evaluating will only be worthwhile if you act on the results. While it can be disappointing to discover that a program is not achieving its intended outcomes, a well-considered evaluation will provide a good starting point for working with colleagues and stakeholders to redesign the program and improve performance.
Dr Gillian McAllister