Meeting of the Commission [Draft]

Meeting date: Monday, June 24, 2024

Official Report 677KB pdf

---

“Quality of public audit in Scotland: Annual report 2023/24”

Agenda item 4 is to take evidence on “Quality of public audit in Scotland: Annual report 2023-24”. This is our final evidence session today. I welcome back from Audit Scotland Alan Alexander, chair of the board; Stephen Boyle, Auditor General for Scotland; and Vicki Bibby, chief operating officer. I also welcome Owen Smith, who is senior manager, audit quality and appointments at Audit Scotland.

We will move straight to questions. From looking at the different themes and the suggestions that have been made, to what extent have the 2022-23 recommendations been actioned, given the decrease in the results that has been identified in 2023-24?

I will ask Owen Smith to set out for the commission what we asked of auditors following the 2022-23 report and the progress that has been made, together with some background information—if you would find it helpful—on how we track and monitor the progress that auditors who have been appointed by me and the Accounts Commission are making in respect of the quality of their work. That is informed by the variety of sources that we use through our arrangements with the Institute of Chartered Accountants in England and Wales to conduct quality assessments and audit providers’ internal quality arrangements, and we triangulate that with information that we get from various stakeholders who are recipients of audits.

I will say one thing before I pass the question to Owen Smith. I think that we expect recommendations for improvement almost every year. The bar for representative audit quality is getting higher—and rightly so. As the commission will have seen over many years, there have been instances—thankfully, largely in the commercial setting—in which there have been deficiencies in audit quality. The standard of what constitutes a high-quality audit is increasing.

In “Quality of public audit in Scotland: Annual report 2023/24”, there are some more examples of where we want auditors to improve. We also ask whether those are the same examples. Are we seeing an audit still not meeting what it should meet? Are the same issues still being identified by reviewers, or are newer or more specific things happening from one year to another?

I will pass over to Owen Smith to say more.

Before Owen Smith gives us information, it might be interesting to note that the audit quality survey results for the audit services group have declined from 2022-23 levels, and they are in a low position compared with those for other audit providers. It might be good to touch on that.

I will bring in Vicki Bibby to say a bit more about the survey results specifically. I omitted to mention that the views of our colleagues are also a key factor in considering how delivering audits feels and what is being picked up in respect of the expectations that Audit Scotland and the other providers set for them.

It is all well and good having rhetoric that says, “You are encouraged to deliver a quality audit,” but the relevant question is whether colleagues feel that they have the right levels of resources and time to do so. That is set out in detail in the report that we are discussing.

If I may, I will first hand over to Owen Smith, and then perhaps Vicki Bibby could come in.

Good morning, members of the commission. Our audit quality framework collects a range of performance or audit quality indicators, including ones that cover staff views. The key areas for improvement referenced in the previous “Quality of public audit in Scotland” report related to the cold reviews carried out by our regulator, which is the Institute of Chartered Accountants in England and Wales.

Every year, we write to each auditor who is affected by those inspections and ask them for an action plan so that we can see what they have done in response. We also ask for what is called a root-cause analysis, which is a detailed review that the audit firm itself will conduct to understand what went wrong and therefore what could be improved. We also encourage that approach for high-scoring inspections, to see what went well in those cases that could be shared across other teams and audits. We get the responses in the form of action plans, which we use to track improvements year on year.

As Stephen Boyle alluded to, the critical test for any improvement will have been whether the finding was repeated at the following year’s inspection. I am pleased to say that, in the majority of the cases that were referenced in this year’s report and last year’s, all the improvements have been actioned and the adverse findings have not been repeated. We take assurance from the fact that audit teams took the recommendations seriously and put in place appropriate responses. When the ICAEW inspects the same audit firms and methodologies in the next year and does not find the same issues, that shows that the firms have been successful in trying to fix those problems.

I will bring in Mark Ruskell.

On the back of that question, I want to drill down into an aspect that staff have reflected on, which is the availability of training to enable them to deliver high-quality audit work.

It is clear that you offer your in-house teams more training days than private firms do. However, when you ask your own staff whether their training level is adequate to deliver such quality, significant numbers of them still feel that it is not. I want to understand what lies behind that. Is it because you have a lot of new staff coming in who feel that they have particularly high training needs? Does it relate to the points that we discussed earlier, about the need for peer-to-peer support?

It is a little concerning, so I would like to understand how you are addressing your staff’s perception that, even though their training provision is good, as regards the high number of days that are available, it does not seem to give significant numbers of them the confidence that they need to deliver the highest quality of audit work.

I ask Vicki Bibby to come in on that.

I will also address your comment about the wider staff survey, because those aspects are all interrelated. That links to our earlier discussion about balancing recovery and getting back into audit timelines with staff wellbeing. It also relates to last year being the first year of the new audit cycle, which means that staff who go into new organisations have to build up. Even without the added pressure of recovery, the first year of an audit always brings extra pressure. We do a number of engagement surveys with our staff and the message that we hear back from them is consistent as regards balancing pressure and wellbeing. We take that very seriously.

I will turn to training in a moment. However, your point also links to the subject of our next session, which will be audit modernisation at Audit Scotland. I will not go into the detail of what we will talk about then, but I will say that our staff tell us that our systems are not fit for purpose in the modern audit sector, which they feel holds them back. At our next session, we will explain in more detail why we need to modernise our audit approach and the systems that are required to support that.

In addition to training, a number of accounting standards have come out. Our innovation and quality team is working closely with the audit services group to ensure that staff who are deployed on audits get the guidance and information that they need to apply the new accounting standards. It is about the timing of that, as some audits might be early, with some of the information still working its way through. Our innovation and quality team and the audit services group are actively working on that, and the executive directors are working closely on it, too.

The point about training relates more to specific technical issues coming up and the timing of those, rather than the general training that we provide for staff. However, it is absolutely right that the training must be relevant and timely to be effective.

That partially explains things, but I still do not fully understand the differences with the firms. Do staff in the firms not come up against those issues?

The table just below paragraph 36 of the report shows a slight clustering of those who responded to the statement:

“The training and development I receive enables a high quality audit”.

The table analyses the trend over a five-year period, and the downward movement from 2021-22 to 2022-23 can largely be attributed to, as Vicki Bibby referred to, the introduction of our very complex auditing standard 315 and the assessment of the control environment when using digital tools and techniques. Colleagues have told us that it took a bit of time to get that right. Vicki Bibby is right that we have deployed a variety of training, seminars and materials to try to give colleagues the right level of insight and confidence to do the work, but we needed a further year. The uptick in the 2023-24 results shows that the approach has become more embedded, with the training being seen to be of value.

It is also about dialogue, because training is not done in isolation. The provider does not say, “Here you go—you’re now equipped.” We have to base things on feedback. Did the training deliver what was intended? Do people feel that they have the skills to do the work?

Vicki Bibby is right that there are other factors. We have new colleagues. Our turnover is higher than it has been in previous years. Such variables will all influence the situation.

In my role not only as the accountable officer but as the AGS, I take confidence when appointing auditors from getting feedback that shows that we are back up to a figure of 60 per cent or so on that issue in Audit Scotland this year. I am hopeful that the divergence will end as Audit Scotland gets back in the pack with other providers.

However, it is a tough environment. As Owen Smith mentioned, the auditing standards are becoming more regulated, and there is also the complexity of accounting standards for non-current assets. Regulators such as the Financial Reporting Council and accountancy bodies take a view on whether auditors are doing things properly. That creates a degree of anxiety among members of staff about whether they are equipped. It will take a period of time before we satisfy existing standards, and there is an expectation that there will be more to follow.

I understand. Thank you.

I want to pick up on a few issues that have come up, particularly the variation between in-house auditing and that done by external auditors. Do you have any statistical information that could allow us to compare performance or completion against targets, for example? That would give us a feel for whether external auditors are more on target than your internal teams, or not, as the case may be.

I will ask colleagues to provide clarification.

In our report, we made a broad assessment of the quality of public audit in Scotland. We did that in a number of ways. For example, we considered the timeliness of the delivery of the work, and we did a retrospective analysis of the quality of judgments in relation to whether they were supported by the right evidence, documentation and so on. That is set out in the table at paragraph 71 of the report, which shows how the various providers are performing based on the assessment of the regulatory bodies.

I am happy to comment in whichever way you prefer, deputy chair. Is it about the timeliness of completion or is it more generally about the overall quality of compliance?

I ask you to stick to the timeliness of completion, as I have some other questions about that.

In that case, I will hand over to Vicki Bibby and Owen Smith.

12:00  

In the first evidence session this morning, Martin Walker highlighted where we had got to as of last week. I think that the figures were 99 per cent for Audit Scotland and 83 per cent for the firms. The mixed model gives us data that we can use to compare the two. In this instance, we are performing better. It is not a competition, but we do not feel that there is any risk that we are not performing in line with the firms. In relation specifically to delivery, we are at a higher position than the firms.

Presumably, it is the case that some of that work is blended anyway, with some of it being done by external auditors and some by your staff. Do they work together in that way, or are they very distinct in that respect?

In some ways, it is both. I and the Accounts Commission appoint auditors from firms to audit the accounts of public bodies as part of the annual audit process, as well as named colleagues at Audit Scotland to be the appointed auditors. They deliver the annual audit with their teams and then, as you have just heard from Audit Scotland’s auditors, they do something similar: they give an opinion on the annual report and accounts and they sign that in their own names as the appointed auditors. However, that is not to say that we do not work collectively; we do that through what we call sectoral arrangements. For example, the auditors of the NHS work in a collegiate way to share intelligence and support our public reporting work.

Through some of the work that we do in this first year under the new code of audit practice, you will see the results of our integrated best value auditing, and that, too, requires close co-ordination and discussion between Audit Scotland’s performance audit and best value team and the annual auditors. You will see that through the reporting that the controller of audit provides to the Accounts Commission. Where appropriate, auditors regularly come together to share intelligence so that we can maximise our audit work to best effect.

That is helpful—thank you.

I am looking at the audit performance in terms of audits completed on time on a sectoral basis—we have touched on that, and you gave us an indication of some of the reasons behind it—and I note that there has been quite a stark drop over the past couple of years. If we look back to 2018-19, nearly every major sector was delivering audits on time. Looking at the chart, I guess that the figures were north of 95 per cent across local government, the NHS, central Government and the FE sector. In 2022-23, however, the numbers dropped considerably, with some areas performing better than others. The NHS was sitting at 74 per cent, at the high end of the spectrum, while the figure for local government was as low as 29 per cent. That is a stark difference.

Will you expand on what has happened over the past couple of years? Obviously, the pandemic is the number 1 factor and the buzzword there, but there was a drop in 2019-20, which was before the pandemic really hit. The numbers were already starting to fall in relation to the targets. I give you an opportunity to expand on that a little bit.

I will ask Owen Smith to set out a bit more detail and drill into the point that you make.

There are variances by sector. We touched on that earlier when we discussed the prioritisation routes that auditors need to take for a variety of reasons where the delay is their responsibility. Under the consolidation arrangements that exist for some Scottish public sector accounts, there is prioritisation of pension fund reporting, because that has an impact on other accounting. However, some of the delays are attributable to the need for completeness in the preparation of accounts.

We are seeing capacity issues in some public sector bodies. Martin Walker mentioned that we are approaching completion of all the 2022-23 annual reports and accounts. Bearing in mind that we are now in 2024-25, that illustrates the fact that there have been significant delays.

You mentioned further education. I am still awaiting the completion of some audits from FE institutions in order to allow me to consider any statutory reporting on that sector. Historically, that would have been done much earlier than it has been.

The impact of delays is real, and it causes a degree of frustration for auditors, for me, as the recipient of those audits, and for the audited bodies.

I will pass over to Owen Smith to say more about the recovery and some of the underlying trends.

The deputy chair is right in what he said about 2019-20, but we changed the target dates at the start of the pandemic, to give auditors and bodies more time to prepare their accounts. I cannot remember exactly, but that whole period was a period of shifting sand, in that the dates kept moving.

We will report on this year’s audits next year. For the first time in a long time, we have had consistent dates for audit completion, which means that we have a basis for comparison. We have set out to all the auditors our expectations as regards improvement. As a minimum, we want audits to be signed off in an 11-month period, or a month earlier for those bodies whose audits are already late. All the auditors have plans for when they will deliver their audits. The fact that they have a broad portfolio was discussed earlier. They will start with the NHS, before moving on to local government and so on. Those teams are all prepared to move through the audits, but if one domino falls, that can have a knock-on impact on that planning. The NHS comes out best, because everyone is doing that audit work right now. There are no complications, other than prior year late audits.

The important thing for me, in my role in managing the contracts, the performance and the reporting on quality, is that auditors have plans in place to deliver improvement, against which we can hold them to account. That is particularly important this year or next year, as we have consistent dates. The dates that we report are a cut-off in time—it is almost like a pass or a fail. Some of the information that was given earlier relates to how many audits were completed within, say, two months, which is more interesting. We are going to bring in more nuanced information, such as sub-key performance indicators, in an effort to capture that better and to understand what has happened.

Some of the bodies for which this year’s audits were late were also late the year before. It is very hard to meet a target date on time when the previous audit was six months late. We need to see a recovery over the next two, three or four years, and that is what we are really invested in at the moment.

Do you get the impression that the issue is that the organisations in question simply do not have the capacity, or that they do not take the issue seriously enough? In some sectors, the delays are quite stark. Given that a huge amount of public money goes into those bodies, that raises eyebrows, if nothing else.

We absolutely recognise that timeliness is a component of audit quality. An aspect of the quality of individual accountable officers is that, when they are appointed, they will be able to complete their accounts on time and to the highest standard, to support parliamentary and public understanding of how public money was spent. That is at the heart of the accountable officer role and the responsibilities that we have that go alongside that.

I am not detecting any unwillingness or a blasé approach on the part of public bodies about the importance of the preparation of accounts. That is not the feedback that we are getting. We know from the feedback that we get from all the public bodies that we deal with that they want their audits to be completed to timescales, to support their governance arrangements and the workings of their organisations.

To go back to our earlier discussion, the question is who it would be best to be responsible for that. Perhaps understandably, some public bodies have an expectation that the auditor will be able to support the preparation of the accounts to ensure that that process is seen through more swiftly. However, auditors are clear in their view that they cannot bridge the independence of their role. It is for public bodies to prepare their accounts.

As you mentioned earlier, it is important that public bodies have the right level of support and training in that regard. More than 200 sets of accounts need to be prepared and audited.

In some cases, we are seeing that there is a limited pool of people who are best placed to do that, and the support that larger sector leaders are providing is inconsistent in that regard. There is work to be done to support the recovery of the shared endeavour of having a completed audited set of accounts.

You have a separate measure of audit quality, which involves your appointment of the Institute of Chartered Accountants in England and Wales to carry out performance reviews on a sample of—in this case—eight audits. If I interpret the findings correctly, the ICAEW said that only four of the eight audits of which it carried out a performance review met the standard that was expected. Is that your reading of the findings? If so, what conclusions do you draw from that?

You are right that Audit Scotland, on behalf of myself and the Accounts Commission, appointed the ICAEW to act as an external assessor to assess the quality of audit work that was carried out by appointed auditors through the annual audit process. As Owen Smith mentioned, our audit quality framework also extends into our performance audit and best value work.

ICAEW scores individual audits on what is known as a cold review basis, and uses the Financial Reporting Council’s scoring mechanism, where audits are ranked on a score of 1 to 4, with 1 suggesting that no improvements are required and 2 suggesting that minor improvements are required. That is the cut-off point that I and the Accounts Commission want. A score of 3 suggests that improvements are required, and a score of 4—thankfully, we have none—suggests that significant improvements are required.

I have a couple of points to highlight for the commission. Some audits were scored a grade 4 in 2019-20 and 2020-21, and I, the commission and the Audit Scotland board were clear that that was unacceptable—that is not a position that we wanted to be in with regard to our ability to offer public and parliamentary assurance about the quality of audit work.

We are happy that there are now no grade 4 audits. Three of the audits that got a score of 3 this year related to the work of the audit services group and one related to Forvis Mazars Accountants, which carried out one of the audits through the audit framework.

Where there is a score of 3 or 4, it results in engagement from Owen Smith and his colleagues—Owen might want to say a bit more about the root-cause analysis work and the improvement plan that we ask for, and then perhaps step back for a second to address the issue of whether there are any trends or recurring issues in relation to the improvement action plan that the audit quality and appointments team requests. We take audit quality incredibly seriously. It matters in terms of the credibility of audit work and the reliability that can be placed on it.

There are a number of benchmarks with which to assess whether an audit was done properly, but the most fundamental is whether the audit opinion was sound. When an auditor puts their name to an audit opinion, that can be seen as reliable. The audit opinion is the judgment that they make about whether the accounts on which they giving their opinion contained any material misstatements. Thankfully, when ICAEW has made the assessment that some improvements are required, it has not cast doubt on the reliability of the auditor’s opinion. However, as I mentioned, there is work to do in relation to audits that received a score of 3 this year.

We contract with ICAEW to do those reviews in order to ensure absolute independence. However much I am independent from the auditors, I still work for Audit Scotland, so it is important that we have a regulator that is approved by the FRC to do that work. Importantly, as part of that inspection process—that is what the ICAEW calls it—the auditor, subject to review, will have to prepare an action plan in response to the findings, and we follow that through, as I indicated earlier. The table at paragraph 66—which gets me in trouble because it is quite complicated—tries to expand the sample of all the different types of reviews that we undertake in order to try to get a handle on quality.

It goes back three years to give a cumulative target. This year’s results show that only one of the different cold reviews that are undertaken did not meet the target. Those are independent ICAEW reviews. The picture has improved quite a lot since 2021, which is one of the times that we started to do this.

12:15  

There is a direction of travel over time, which we are pleased to see. As Stephen Boyle said, there are few repeat findings, so the regulator’s focus will move over time, as auditing standards change. It is a moveable feast. We will always get recommendations, which is why we have a comprehensive audit quality framework to find out what needs to be improved and to take action to make that happen.

The other side of the AQF is to get assurance for the Audit General and the Accounts Commission that the people who they audit are of a certain standard. I think that the direction of travel is the right one—the figures are going up.

Can I just probe that a little bit more? My reading is that the good compliance record was much higher the year before—the percentage was in the 80s. Is that evidence of things going in the right direction, or is it evidence of things going in the wrong direction?

That is why I referred back to the previous table, which gives an average over three years and gives a wider base to come to that conclusion. The in-year results are important, but the year-to-year results show a drop. However, if you go back over the last three-year rolling period, it shows improvement over time.

It feels a little bit like you might be manipulating the figures, Mr Smith.

No. I do not score those audits. An independent person does that.

It is pretty plain that no one wants a visit from you any time soon, is it not? Things have gone wrong if you are knocking on their door.

Maybe I can add a couple of points. Owen Smith’s point is fair. We want to make an assessment of the direction of travel of audit quality in both our financial audits and our performance audit and best value work.

On the latter, the assessment that ICAEW has made, and which is also informed by the internal cold review work that our innovation and quality department will do on performance audit and best value, is that the audits are consistently good. That means that the performance audit reporting, the briefings and the performance audits that I take to the Public Audit Committee and that the Accounts Commission receive meet the standards consistently. That gives me a lot of assurance that the work that I am presenting to the Public Audit Committee is reliable.

On the annual audits, there is a volume element. We are auditing more than 200 sets of accounts. We are appointing six firms and a significant number of Audit Scotland colleagues to audit individual public bodies. We are applying the learning that we get from one round of quality findings to the next. Having said that, there is always the possibility that an audit does not meet the required standard for whatever reason. You see that either through a documentation issue in which an auditor has not kept on their files the evidence with which they are making a judgment, or, more fundamentally, through their issuing an opinion that is unsound.

We are not in that latter category. That also gives me assurance that, although there is some variability, the underpinnings through the audit quality framework are right, that there is follow-up in terms of root-cause analysis, sharing good practice and learning where things have not gone right, and that we follow through and track the recommendations in the action plans.

To add to that, the executive team and the boards get the update report that Stephen Boyle referred to. We have already had the update report from the executive director of innovation and quality that the root causes of where the scores of 3 were identified this year have already been addressed in our audit approach.

The report says that there are examples of insufficient “challenge of management”. That has not affected the outcome of the assurance around the audit; it is just a methodology thing. Is that what is being said there?

I will pass that over to Alan Alexander in a moment. It is important to say that there was insufficient evidence on the file. This is what ICAEW will be making its assessment of, as will our in-house quality team. A mantra is drilled into auditors: if it is not on the file, it did not happen. There has to be rigorous record keeping.

As we have heard, for a variety of reasons, that did not always happen in every audit. Often, when cold reviewers come in at a later stage, they say, “Okay, I can understand why you’ve made that judgment and have accepted it, and you might have documented that you took assurance from management or that you challenged them, but I cannot see the evidence of that on the file.” If that evidence is not there, that calls into question whether management assurances should have been adequately relied on.

The regulatory regime for auditors and accountants in the UK is strengthening. Although we do not yet have a transition from the Financial Reporting Council to the new planned regulator—the audit, reporting and governance authority—it is clear that the Financial Reporting Council is placing more emphasis on auditors’ reliance on other experts, whether they are management experts or auditor experts. That is absolutely central to the disclosure in balance sheets of non-current assets, fixed assets, equipment and so on, because the regulator, through the FRC, wants more evidence on how assets are being valued and on their ownership. That must play through to the work of auditors.

I hope that I have reassured the commission that we take the issue seriously and that we are responding to it through training and development, because I know that the firms that we appoint will want that level of evidence on their files before they certify their accounts. It is a complex and moving environment.

I will bring in Alan Alexander to say a bit about how our board is dealing with the issue.

It is a sensitive and complex issue for an independent board. I am quasi-religious about the distinction between governance and management. We are there to provide good governance and scrutiny.

When I became the chair of the board in April 2020, I said to the then chief operating officer and the then Auditor General that I had one requirement: I wanted no surprises. Even if I would not be getting involved in an issue because it was a management matter, I wanted to know about it.

The first call that I got was about an unexpected drop in audit quality. My first question was, “What are we going to do about it?” One of the things that we did was to improve the quality and detail of reporting to the board so that, if the trajectory was not right, we could intervene at any point. The papers that the board gets are much more multitextured than they were in my first 18 months on the board. That allows us not only to say, “That’s not good enough”—I am being brief deliberately—but to say, “These are the areas that we think you should be spending more energy on.”

The result of that is that the board is now happier about the quality of public audit and the trajectory. We have managed to maintain a pretty impermeable Chinese wall between the audit quality and appointments team and the rest of the organisation, and the ICAEW is doing absolutely independent reviews. That has given the board confidence that the information is good and, therefore, confidence about any interventions that might wish to be made in areas that need closer attention.

I am content that the quality of public audit in Scotland is now at a generally higher level than it was at four years ago, but, like anything that we assess in this way, the last bits of improvement are always the most difficult to accomplish. The board is on the case, but it is very careful to not get into the nitty-gritty. One of the things that gives us confidence in that regard is the fact that there are two levels of independent scrutiny of our quality.

Thank you very much.

As there are no other questions from members, I will bring the meeting to a close. In doing so, I acknowledge that this is the last meeting that Professor Alan Alexander will attend as chair of the board. I thank him for his significant contribution over a number of years.

That is very generous of you, but you do not get rid of me that easily—I will be here on 30 September, albeit not in a formal meeting of the commission. It has been an honour and a pleasure to do this job. One of the impressive things for me is that, even at my great age and experience, I have learned a lot of things over the past four years that I wish that I had known when I was trying to improve the management of local authorities 25 or 30 years ago. Thank you very much.

Thank you.

Meeting closed at 12:25.