Scottish Commission for Public Audit, 03 Dec 2008

Meeting date: Wednesday, December 3, 2008

Official Report 110KB pdf

---

Economy, Efficiency and Effectiveness Examination of Audit Scotland

Our next item is a follow-up review of a three Es study that was undertaken by HW chartered accountants in 2006. The commission instructed the review earlier this year to find out what progress was being made on Audit Scotland's approach to fees and charges.

I extend a warm welcome to Jay Hussain, a partner in HW chartered accountants, who is here to speak to the report and answer any questions that members may have. I invite him to say a few words on the follow-up study.

Good morning, convener. As ever, it is a pleasure to be here. Thank you for inviting me.

The year has flown by; only a few months ago we were sitting here talking about a follow-up review. The commission instructed us to have a high-level look at the progress that Audit Scotland had made against eight key recommendations in the 2006 review of fees and charges. As I said at the time, that was an expansive report. Although it was titled "Fees and Charges", it covered everything below, above and around that, which is why the recommendations that we made then were wide ranging and far reaching. It may be useful quickly to remind members of the 2006 report's key messages. The review came about because members wished to learn a bit more about what the audit fee was, how it was made up, how transparent it was and so on.

We reported back to the commission that the audit fee was made up of two elements: first, a fee that was agreed directly with the audited body; and secondly, a fixed charge. The latter element was interesting. We recommended that Audit Scotland could make the fixed charge more transparent by explaining to audited bodies what it did and did not include. On that front, I am pleased to say that the fee letter to an audited body now conveys in much more detail what the fixed charge element of the fee relates to. It is made explicit that the fixed charge includes an element of overhead for running Audit Scotland. That aspect alone is important because audited bodies are now clear that part of the fixed charge contributes to the cost of running Audit Scotland. That is as it should be and is fine. We are therefore comfortable that Audit Scotland has done what it needs to do on that key recommendation. I stress, however, that that is not the same as debating whether the fixed charge adds value.

Beneath that, other, wider issues came out about whether internal cost management was transparent. Audit Scotland's service delivery arms were managing time, but we asked whether they were aware of the costs of delivering pieces of work and so on. We also reported on that.

Covering that in the round was performance management and the performance management system. In that respect, we asked what Audit Scotland was about and how we could measure its efficiency and effectiveness. We talked in some detail in 2006 about Audit Scotland perhaps introducing more overt efficiency and effectiveness targets, and we gave a few examples of what they might be.

Since we reported on those themes in 2006, we have had a range of meetings with a number of people in Audit Scotland and we are generally satisfied with its progress against the eight recommendations. Members will have seen in the appendix to the follow-up study that many actions are on-going or partly completed. That is not entirely surprising because the issues were wide ranging and it will take some time to get everything into place.

I have jotted down some of the good things that have been done. As I said, Audit Scotland was asked to make the fixed fee clear. That is much more transparent now.

There is a great deal of self-awareness about the need to be more vigilant and tight on the cost of actions. Indeed, Audit Scotland commissioned its own internal best-value review, which was a detailed nuts-and-bolts one that made many recommendations for the audit services group. We asked whether that should be extended to the other service arms, notably the public reporting group. It was acknowledged that that will happen in time. We hope that extending that review across the public reporting group will prove useful because the ASG and the PRG are the two main service delivery arms.

As for the foundation to all that, when we last met, we spoke about the time recording system. It is obvious that we need to understand costs. To do that, we need to have an engine to feed things into. We talked about that in our report a couple of years ago. In April 2007, a time recording system was put in place. We have examined that, as have the internal auditors. The system is being used in the audit services group. The reporting capabilities exist; I have looked at them and the system is pretty user-friendly. The issue now is how quickly the use of the system can be rolled out to achieve commonality in what it produces for all Audit Scotland's service delivery arms.

The audit services group is using the time recording system, but the public reporting group is not. I have spoken to the PRG, which recognises that it needs to implement the system fairly quickly. The PRG's involvement is important because the fixed charge that I mentioned a few minutes ago covers much of its work—the national studies. Using the system is important, but it is arguably more important that the PRG uses it more robustly, because finding the audit trail from the impact of national studies to local bodies is more difficult. Work is being done on that in the health sector, which is great. However, rolling out the time recording system fairly quickly is an important development. Audit Scotland acknowledges that, but a timeframe needs to be in place to ensure that that is happening.

The system has the bells and whistles. For reporting, it can generate all levels of information. The internal audit did not raise significant issues with its use and it echoed our comment that the system needs to be used in the public reporting group much more.

Paragraph 2.4 of the executive summary encapsulates what I have said. The second bullet point in that paragraph is all about using the time recording system and the information from it. The third bullet point follows that. We sensed a need—certainly in the public reporting group—for more training on using that system and the financial management system, which is also new. The first, fourth and fifth bullet points are about the same issue—what Audit Scotland does, how it goes about doing that and how it scopes its work. We and Audit Scotland feel that efficiencies can be found and that work can be delivered much more effectively.

I understand that a new corporate plan is in the offing as we speak, which goes to the heart of the issue. Audit Scotland does not just scrutinise bodies and ask whether they are following rules; it helps bodies to improve how they deliver their services. Therein lies the added value of Audit Scotland. We have been told that the corporate plan will include a raft of actions with associated performance indicators, some of which should belong to the Scottish Commission for Public Audit, whereby we will finally have a way of measuring the extent to which Audit Scotland holds public bodies to account and helps them to improve services.

I am happy to take questions.

Thank you for your verbal report and for your work to produce the follow-up document. It is pleasing that you report good progress. I am sure that my colleagues have questions.

You said that the time recording system was recognised to be valuable and useful and that the public reporting group recognised that, too. Given that, why is that group not using the system and what are the implications of that?

The time recording system came in in 2007 and there are two issues to discuss. First, the mechanics of using the system must be understood and I think that some training was rolled out originally. My understanding is that these things always go through the audit services group first, because it is a bit further down the line, before cascading through to the other service delivery arms.

The more important and interesting point, is that there might be a cultural issue—I use the word inculcate in the executive summary. The PRG has, quite rightly, taken a zero tolerance approach when it comes to the accuracy of national studies. Because of the high-profile nature of such work, getting it right has been paramount, however long it takes. To break away from that culture of doing business was always going to take slightly longer than in the audit services group, which was already doing cost management, even before the time recording system came in.

I cannot offer the commission anything more concrete than that. I have spoken to the director of the PRG and it is encouraging that she is fully aware of the need to go down this route and to do so quickly. I suggest that that should happen within the next few months, not years. We suggest training because there is still a knowledge gap there. There has been a lot of confusion between what the time recording system generates and what the financial management system generates. The time recording system will feed the financial management system, and that will generate all the costing information. We need to bridge those fundamental gaps first of all.

I could be wrong, but my understanding was that no formal, tiered training was rolled out at the time; it was pretty much a rolling programme of online training. There were no formal training packages or anything like that. The time recording system is designed to be fairly user-friendly. I have looked at it and it is quite intuitive. I can conclude only that this is more of a cultural issue and that it is about breaking away from not doing cost-management and not setting a budget at the outset. However, that might need to be some sort of added stimulus.

You indicate in your report that the time recording system was introduced as long ago as April 2007, so it has had 18 months to bed in. You use the interesting phrase that

"the use of the TRS continues to bed in within all of Audit Scotland's service areas".

That sounds like a euphemism for saying that it has not bedded in within a significant part of Audit Scotland's mechanisms. It sounds as if management needs to get a grip of this issue and, as you suggest, set time limits and support mechanisms for the system. What you are effectively saying is that the money has been spent on the system and that it is providing only half the value that it should do.

Presumably, the time recording system can feed into pricing—so that there is a link between the work done and the cost that is going out to the various clients—and into the financial management reporting. It may be that it is working okay on the second of those fronts but that it is not really producing anything on the first. Starting now, how long should it reasonably take to get the system going? It seems a bit unsatisfactory that after 18 months it has still not been fully implemented.

I appreciate your sentiments. The second point is a lot easier to answer than the more complex first point. The time recording system can generate a range of financial management reports, which is good. At a high level, Audit Scotland has always had a fairly good grasp of total costs and so on. The challenge, which we reported in some detail two years ago, was, "That's great, but to be an efficient organisation that has to happen all the way down to business unit and, indeed, people level." That is why we are talking about the time recording system. It is the foundation for faciliatating that.

You suggested that our comment on the time recording system was a euphemism. To be absolutely direct, I make it clear that the TRS is being used in one arm but not in another arm. It is starting to be used in the PRG at a high level, but it has yet to be cascaded to assignment level, as it were. When it comes to the use of the TRS for financial management reporting, everything is there—the internal audit review testifies to that—and I have seen examples.

Your question about linkages to fees is more complex. Theoretically, the point of using the TRS is to help to set the selling price, as one does when one sells a product, but the process is obviously a bit more complex for Audit Scotland, given its financial make-up and the fact that it has to break even every year. In addition—I will not labour the point too much, because we could be here all day—the setting of fees is still very much based on notional days. That came about when the organisation was set up some years ago.

In the report, we say that there is a tension between that approach and a risk-based approach to fee setting. Risk-based fee setting has a direct link with how much time is spent auditing an organisation. Risk-based auditing involves risk assessing an organisation at the start of the audit year, building up a programme of work and assigning costs to it. That is a sensible approach, and it is what one would expect to happen. Audit Scotland is going down that road, but the fee-setting mechanism is still grounded in the concept of notional days, whereby an authority that has a net annual expenditure of £100,000 will have 50 notional days attached to its audit and one that has a net annual expenditure of £200,000 will have 100 notional days attached to it.

However, the organisation that spends £100,000 a year could have numerous problems that require more audit effort, with the result that those 50 days are not enough. As a risk-based auditing mechanism involves starting with a blank page, it would be possible to identify 10 fundamental risks, as well as the fact that the organisation had had a case of fraud the previous year, so more time would need to be spent auditing it—the equivalent of about 100 days. Even though the other organisation spends £200,000 a year, the fact that it had had a clean audit every year for the past 10 years would suggest that nothing was going on. If it had a stable management system, it might be possible to knock the audit out in 50 days flat. Whether an organisation spends £100,000 or £1 million, a statutory audit is a statutory audit—the same processes still have to be gone through. Theoretically, the same hours will be spent; it is just that the values will be different.

On your first question, Audit Scotland's plan is to see out the next round of audit appointments. The timeframe is set in the internal best-value review. A risk-based framework for audit fee setting is being considered. All that we can say is that we fully support that approach. That is what we said two years ago, which is why the recommendation came about.

Once everything is bedded in, if a framework for risk-assessed fee setting is in place come the next round of audit appointments, that is when the use of the time recording system to build up costs will really come into its own.

If I understand you correctly, you are saying that, regardless of its potential, the time recording system is not being used to any great extent for the setting of fees.

No, it is not.

In your last report, you said:

"a greater level of rigour in financial management, in particular monitoring of costs in addition to monitoring of time, should be introduced".

On page 6 of the appendix to the follow-up, you say:

"The use of these reports and the culture of ‘cost management' however remains variable although improving."

Again, that rather suggests that the time reporting system is not being used as well as it could be on the reporting information side, either. Is that unfair?

With any system, one can always put forward the argument that it is not being used to its full capacity. Perversely, we almost would not expect the TRS to be fully used at this stage, because there is no direct link to fee setting. We need to go back to basics now regarding the cultural issue that I spoke about earlier. As we look forward to the new corporate plan, there is a clear target in it for efficiency. Efficiency can be achieved only if we know how much things cost.

Let us park the link to fees for a moment. We need the system to be fully used for the operational and strategic management of Audit Scotland. On that point, I agree with your assertion that it is not being fully used. I would be feeling much more comfortable if the public reporting group were using the system a bit more than it is. The internal best-value review has been pretty honest about what is not in place. The whole thing about needing to use the time recording system is a recognised issue internally.

As the external auditors, we want to look forward. We have talked the talk, but let us now put timeframes in place, set up a training programme and address any knowledge gaps that exist. When the auditor is back up for reappointment in three years, let us take a good look at fee setting, from ground level upwards, and move away from the concept of notional days, if that is practical. People we speak to, including the partners of the audit supplier firms, the director of audit strategy and the public reporting group all recognise that the way in which we set the fee should be more overtly risk based.

We are precluded from saying too much. The internal best-value review has recognised what needs to be done, and I would like a clear series of actions to be taking place against those recommendations in a year's time.

I will continue with the subject of the time recording system, and ask you about a couple of aspects. First, you spoke about there being a cultural issue around not using a time recording system. There is a world of difference here. As I know from personal experience, time recording is a hassle if you have to record every six minutes of your day when you work on 10 to 15 different things day by day as a matter of course. On the other hand, if someone works on one project for a sustained period, time recording is not that much of a hassle—it can be a daily or weekly task. As far as the PRG is concerned, how onerous would it be for individuals to record their time?

Secondly, in light of what you have seen in the organisation, what obstacles exist that would prevent a move to full recording within the PRG? What timescale would be required? Would it be realistic to suggest that the organisation should move to using the system within three months? Would that be feasible or not?

My final point is a fundamental one, and it concerns efficiency. If we do not have a baseline or a starting point in relation to Audit Scotland's costs, presumably it will be unable to demonstrate efficiencies, at least until it has a baseline and a comparator. How on earth is the commission meant to take an objective view on whether or not there has been any improvement? Are we left in a position where the only thing that we can do to measure Audit Scotland's efficiency is to take its word for it?

Starting with your first question, on whether the time recording system is onerous, I think that it is not. I have looked at it and I have read the internal audit report on it. Even people who have not fully used it yet, including the PRG, are telling us that it is pretty intuitive and easy to use. I honestly cannot imagine that rolling it out will be onerous. If anything, we were told that the modified system that was introduced in April 2007 is much more tailored to the way in which Audit Scotland operates. The answer is therefore no, the system is not onerous. The issue is embedded in the culture, and the question is, how long is a piece of string? There needs to be management impetus on the time recording system, which there is. The intention to implement it exists, which is all well and good, but perhaps we need a formal timeline.

Your second point was on obstacles around using the time recording system fully. The system is such that whether you are undertaking a national study or an audit of a body, you still put in time against what you have done. Audit Scotland will have codes around part planning, financial statements work or best-value work. There are no real obstacles, nor would we expect there to be, because the system should have been looked at properly before being introduced. I have not looked at it in detail, but, on the face of it, the system looks fairly intuitive, clear and user friendly. When I spoke to the director of PRG health, she showed me a high-level report from the ASG and told me that it contained what she wanted for PRG and that she could do things that way. The coding structure of the time recording system allows you to deal with local authority A, B and C and national study A and project B. It links into the coding structure of the financial management system. All the building blocks are there to be used, so let us get on with it.

I agree entirely with the point about efficiency: it was pretty much the nub of what we were getting at in 2006. I spoke a few minutes ago about high-level and strategic-level efficiency savings. We are fairly comfortable that Audit Scotland knows where it is at. The interesting question is whether there is a clear audit trail between high-level efficiency savings and where they come from. You would expect a strategic-level efficiency saving to be cascaded down to service groups for them to make their savings. My understanding is that that is pretty much at the centre of the radar in Audit Scotland.

An efficiency group is considering that issue, although it has not looked into it in great detail yet. However, our understanding is that its sole aim is to ensure that whatever strategic efficiency target Audit Scotland sets is followed through at a service level, so that everybody is accountable, as you would expect; otherwise people could just come up with a figure at the top and you would not know where it had come from. It is important that you have a clear audit trail between a saving, such as a 3 per cent saving globally, and where it has come from, such as whether it has come from real cash savings or from doing some things a bit better. Another interesting thing about the time recording system is that it will allow you to see how long one person is spending on an activity versus the next guy. All those things are interlinked. I concur with Derek Brownlee. We expect some developments. I think that the efficiency group is on the case in relation to translating the global efficiency target or targets to the service level. That is where the time recording system will kick in.

I want to ask a simplistic, blunt question, which is the wont of politicians. We have eight recommendations, only one of which has been actioned in full; the rest have been actioned in part or are on-going. This is nearly two years on from the original report. Is what has been achieved to date reasonable? When would it be reasonable for the commission to expect more recommendations to be actioned in full?

I try not to get drawn on that sort of question. The recommendations were, almost by their nature, quite wide ranging and there were sub-facets to them, too. I speak from experience of case studies that we have done at the National Audit Office, for example. Ideally, as a rule of thumb, we expect audit recommendations in value-for-money reports to be actioned within 12 to 18 months of the report being issued, but it can take up to two years, depending on the nature of the recommendation.

As we have said up front and centre, Audit Scotland is in the middle of some pretty big changes. With initiatives such as the Crerar review, for example, there has been a big push to streamline the current audit and scrutiny systems, and I feel that, given its statutory responsibilities, Audit Scotland should be at the centre of that activity. Audit Scotland has had to deal with all that—indeed, it has almost had to reposition itself—and I think that the root-and-branch review embodied in the corporate planning process was certainly needed. That review has been accomplished and I am encouraged to learn that the revised corporate plan setting out the actions and targets is imminent.

Convener, I hope that you will tell me if you think that I am skirting around the issue. Given the various recommendations that are being implemented, it does not surprise me that certain activity is on-going. However, if I had to pick one thing on which I wish more had been done, it would have to be the time recording system. I wish that it was being used more fully in the PRG, but then Audit Scotland recognises as much itself.

As I said earlier, there has also been a fairly nuts-and-bolts internal best-value review, which has examined every facet of the audit services group and has made recommendations on, for example, risk assessment, working practices, training, using the time recording system and, basically, how things can be done a lot more efficiently. Those recommendations will also apply to the PRG, and I expect a clear timeline for implementing them to be established. Indeed, I hope that they will introduced in six to 12 months, and certainly in time for the next round of audit appointments when, as we have been told, the issue of fees setting will be re-examined.

So, notwithstanding your comments on the time recording system, you are content with the quantity and quality of progress.

Yes.

You say that there are priorities within all the things that are happening. How central is the time recording system to the risk assessment and target setting activity to which you have referred? If, as your comments and report seem to suggest, it is not possible to do a lot of things until the time recording system is up and running, is that system the first in the hierarchy of priorities?

That is fair comment. For me, the time recording system is the foundation of any approach to proper financial and cost management. As I have said several times, it would have been nice if the system had been rolled out more fully in the PRG.

I listened with interest to your comments on the impact of Audit Scotland's work. Indeed, paragraph 2.3 of your report mentions the

"Development of a framework for measuring the impact of"

Audit Scotland's work. As a lay person, I am interested in finding out how far that framework has developed and how it compares with other models in the audit world.

The work on impacts is well under way. It all stems from the issue of what Audit Scotland was actually getting for its fixed charges, which was raised in the 2006 report. Since then, the PRG's response has been pretty impressive, as it has been under pressure to demonstrate the importance of that national study to local bodies. I will not regurgitate our report for the commission, but the PRG has visited audited bodies and committees in the health sector and given presentations on the important issues that they need to consider.

In the external auditor's annual audit report to bodies, messages are drawn from national studies and their importance to the particular body is noted. That is all good, because it demonstrates to audited bodies why certain reviews are important and what those bodies need to take away from them.

The framework is developed within the PRG, which is right and proper, because the PRG's time is covered by the fixed charge. I am comfortable that things have happened the right way round, in that the PRG has developed the framework first. We say in the report that that is fantastic, and that we should examine what the PRG is doing, and use that framework and consider how we can develop it within the audit services group.

It comes back to commonality, and the need to extend the framework to cover different bodies. To answer your question directly, I point out that the impact framework is developed within the PRG, and it is being developed further through the impact group in the other areas of Audit Scotland. We need to view the process as on-going.

In the report, you mention corporate performance indicators. Can you give some practical examples of the kind of indicators that might be of use to the commission?

I should know that off the top of my head. In the 2006 report, we gave some examples of effectiveness and efficiency indicators, which I know Diane McGiffen has examined. There will be a range of indicators. The indicators that we discussed in 2006 included national reporting of progress; the reasons for slippages, which are important, as they go to the heart of efficiency; the issuing of the annual audit report to members within a predetermined timeframe, which draws on experiences in England and Wales, where we have targets for reporting certain things; the issuing of best-value reports within predetermined timeframes; and progress on following up key recommendations.

One of the big issues in England and Wales has been the concern about letting recommendations languish, so a proper timeframe should be in place. When an external auditor makes recommendations about effectiveness, the auditor should go back in six months and see exactly what the body has done about them.

On the efficiency side, efficiency savings should be measured at a group level, which builds on what I said earlier. It is all well and good to have a strategic efficiency target, but we need to translate that into a group-level target. Absence management, the impact on processes of new information technology such as the time recording system, and periodic reporting of expenditure versus budget per group or per project were also included. In the PRG, for example, assignments should be set a cost budget, not just a time budget, which can then be measured against.

Those are the sorts of things that we headlined two years ago as possible performance measures for use by the SCPA, which I know corporate services has examined. You will have to decide what you think about the indicators.

I see that there are no more questions. I thank Jay Hussain for coming to the meeting and giving such comprehensive evidence. Thank you for your time and the work that you have undertaken—it is much appreciated, and I am sure that my colleagues on the commission have found it very illuminating and helpful.

Thank you very much.

Given that we have discussed time recording systems and performance reporting, do members wish to take evidence from Audit Scotland with regard to the follow-up report in the spring?

We could write to them.

Okay—members appear to be content with that suggestion. I am sure that Mark Brough will action that.

Meeting continued in private until 12:10.