How Can Supreme Audit Institutions Select the Right Topics for Evaluations and Performance Audits? Results from the 2024 Meeting of the INTOSAI Working Group on Evaluation of Public Policies and Programs (WGEPPP)

Authors: Nico Granitzer, Andrea Häuptli & Emmanuel Sangra, INTOSAI Working Group on Evaluation of Public Policies and Programs

Evaluations and performance audits require a robust selection process to ensure success. A thorough and systematic examination of potential performance audit topics allows the identification of those that present the highest risks and offer the greatest potential for improvement respectively. Furthermore, it facilitates determining the optimal timing for results to be integrated into the further development of respective public policies. Consequently, a well-considered selection of evaluation topics is more likely to influence existing public policies positively.

As Supreme Audit Institutions (SAIs), we are consequently faced with several challenges: How can we establish institutional structures and procedures to enable the identification of relevant evaluation ideas? And how can we select the evaluations and the timing that will yield the most significant impact from these ideas?

Against this background, the Working Group on Evaluation of Public Policies and Programs (WGEPPP) convened in June 2024 in Bucharest, Romania. We explored how SAIs’ selection processes generally function and how these can produce optimal evaluation and performance audit topics. In addition to the workshops and discussions, a survey conducted among 22 SAIs prior to the meeting provided valuable insights into their identification and selection processes. The main conclusions are presented below.

Figure 1: Selecting and focusing of Evaluation and Performance Audit Topics at SAIs, Based on Sangra & Crémieux, 2013 (1)

Colleagues and Stakeholders as Key Sources of Ideas for Topics

Overall, the Meeting showed that most member SAIs operate with a high degree of autonomy in identifying and selecting audit topics. 91% of the surveyed SAIs can independently choose the majority or all of their topics. The responsibility for generating audit ideas lies with the SAIs themselves. Thereby, an important prerequisite for the selection of suitable performance topics is a well-stocked pool of ideas. Figure 1 schematically illustrates the process by which topics are identified, selected, and focused. 

Internal suggestions play a particularly relevant role: 95% of responding SAIs reported that auditors can propose topics, and 68% regard these suggestions often or systematically useful, especially when they have multiple years of experience in certain policy fields. Most surveyed SAIs complete these suggestions with a risk assessment, identifying structured risks across all audited entities: such processes are in place at 81% of the responding SAIs, and 64% consider the insights gained useful for topic identification. The meeting highlighted that several SAIs increasingly use data analysis to quantify identified risks. For example, the Supreme Audit Office of Slovakia presented an extensive policy mapping, enabling them to identify deficient policy sectors through international benchmarking, such as multiple indicators regarding healthcare policies.

Figure 2: Most Useful Sources for the Identification of Evaluation Topics, As Ranked by the Participants at the 2024 Meeting of WGEPPP

 The two most relevant external sources for new audit topics have been identified as stakeholders involved in public policies and experts in the respective fields. The attending SAIs employ various methods to gather information from these groups. For instance, many hold regular informal meetings (e.g., Switzerland), whilst in other SAIs, contact with those groups is formalised through Advisory Councils comprising members from professional associations or academia (as seen in Romania and India) or regular surveys (e.g., Slovakia). Additionally, 64% of the surveyed SAIs allow citizens to submit topic suggestions through mechanisms such as whistleblowing or other online messaging facilities. These suggestions are considered sometimes useful for topic identification by 77% of these SAIs. Similarly, topic suggestions from Parliament or Government are possible in 86% of the responding SAIs, yet only deemed helpful by 69%.

One important factor that was mentioned in relation to all sources is the importance of providing feedback to the submitters of ideas (colleagues, citizen etc.). SAIs should let them know to what extent their suggestions have been implemented and taken into account. This helps to maintain the motivation and commitment of contributors.

Clearly Defined Selection Criteria and Strategic Planning Processes

Based on these ideas, the selection and preparation of performance audit topics are carried out to maximise the relevance of the audit. The goal is to define the audit objectives, the corresponding focus, and the execution period.

In general, the discussion at the meeting made it clear that establishing clearly defined selection criteria is crucial. Consequently, 77% of the responding SAIs follow a strategic planning process for topic selection in accordance with ISSAI 300 requirements, and 68% impose established quality criteria for the topic selection. On the other hand, the design of this planning process and the criteria used vary greatly. Some SAIs, such as those in Spain, Brazil and the Philippines set new thematic priorities annually, while others base this on a longer-term, overarching strategy of the SAI.

Various aspects are considered in the criteria, such as the potential for change (e.g., Kenya and Bulgaria), the expected costs of the evaluation (e.g., Lithuania), political sensitivity (e.g., Philippines), or the available capacities and competencies within the SAI (e.g., European Court of Auditors). The organisational setup also varies significantly. In recent years, several SAIs have created dedicated strategy units specifically focused on the preparation of audit topics, such as in Romania, Poland, and the European Court of Auditors.

A particular challenge lies in audit areas that have seldom or never been examined. To address this, 64% of the surveyed SAIs conduct preliminary studies to gain a deeper understanding of the current developments and risks associated with the evaluation subject. This efficient approach aids in making informed decisions regarding the execution and timing of the evaluation.

Optimal Topic Selection Through Continuous Development

In conclusion, despite organisational differences and varied approaches, there are significant commonalities among SAIs at a strategic level: practically all SAIs follow a formalised process for identifying and selecting evaluation topics. The establishment of networks with stakeholders and the academic community largely plays an important role, and most SAIs have implemented a strategic planning process. Overall, however, the most essential source of topics is still seen to a large extent as being the auditors themselves. Continuous developments, such as the integration of new data analysis methods and formats for involving external actors, are an ongoing process. In doing so, the increasing relevance of these considerations and the creation of specialised departments within SAIs underscores the growing importance of how to optimally select audit topics.