What is an appropriate way to select a sample for compliance testing?

Following on from the previous article about materiality in compliance engagements - we now look at ways to select an appropriate sample - how large and the methods of selection.

Comment 1:

Sample size in a Compliance engagement really depends on the effectiveness of key controls. I use three level approach - for an entity with good controls he would test 5% of transactions, with medium level assessment of controls say 20%, and poor or no controls (as with a very small sole trader type agency with only a few transactions) I would be looking at 100%. And the sample would not be on a cumulative monetary basis but would be based on say every xth transaction. The large audit firm I worked for had a policy that in compliance testing they would test no more than 25 transactions over a year. 

Comment 2: 

I agree entirely that the sample size will depend on the assessed effectiveness of controls. And the percentages above sit comfortably with me, too.

I do not agree with a maximum sample size of 25. There are probably still plenty of REA trust accounts entirely maintained by the same single individual who does the selling. Mind you, such agencies probably don’t have very many transactions in the sales trust account. If the rental account is audited, where the risks are much greater, there could quite quickly be hundreds of transactions with no effective controls and 100% testing, or very near it, is justified.

Comment 3: 

I actually think it’s illogical to say that sample size depends on effectiveness of controls, because until you’ve tested controls (which you need to have determined a sample size for) you won’t know how effective they are! When I was in practice we used the below for controls testing:

There are a few ways to select the sample, as you say – every xth occurrence, or you can use a random number generator to select occurrences. But not all compliance engagements are also engagements on controls.

Commentary: 

The first comment (from a user) began the conversation and formed the question I asked of the other audit specialists. The second and third comments are talking about slightly different things, depending on the approach taken to gathering evidence. The standards give little guidance apart from general statements such as:

The assurance practitioner shall obtain sufficient appropriate evidence on which to base the opinion, having regard to whether the compliance engagement is a reasonable assurance engagement or a limited assurance engagement. (SAE 3100, 37)

So the actual way that the auditor assesses controls and then decides on how many items to test is really left to their professional judgement. And of course the extent and kind of testing will depend on whether the compliance engagement is reasonable assurance or limited assurance.

In a regulation-based compliance engagement (such as a Real Estate Trust Account) a logical approach would be that the auditor would identify what controls exist to enable the entity to comply with the Regulations. A reputable, well managed group of agencies should have consistent policies and good supervision, and few walk-through tests would confirm that these control policies appear to have been followed. 

In this case - with a preliminary assessment that controls are good - the auditor could decide to test say 5% of transactions. This would of course depend on the control being tested. And this is where the table above comes in. If the control was say "Bank reconciliations are prepared monthly and checked and initialled by the agency licensee" - then it would make sense to use the guidance in the table above and test 2-4 monthly bank reconciliations. If the control was daily - say "banking is checked to receipts daily by a different person than the cashier" then the 20-40 range above would make sense. For multiple items such as receipts the 25-60 range would seem reasonable. 

I think that comment 3 is perhaps in the case where no walk-though testing was done so there is no real knowledge of the effectiveness of controls until detailed transaction testing or controls testing is done. I think a common-sense approach that fits with best practice is that if the testing that is done indicates that controls are poor and errors or issues of non-compliance are happening; that planning would be revisited and testing would be extended to an appropriate level until the auditor was satisfied that they could make the right conclusion or opinion.

Comment 4 is a good common sense observation that I would concur with - having visited a lot of firms to do training: in our attempts to be conservative and minimise risk we set materiality low, then we apply performance materiality at a conservative level, then we adjust our confidence levels to test more transactions because we have concerns about controls, and the result is that we do far too much transaction testing that proves little and blows out our budgets.

A better approach in my view is firstly to look at controls with more care and ensure we document well any good controls, so that we can justify placing confidence on them to reduce the level of substantive testing, focus our testing on any remaining areas of risk, while responding by increasing sample size if we find problems. Then when we do our substantive testing sample take a "common sense" look at the sample size and ask "is the level of testing we are doing here really adding anything to our conclusion? The table above provides this kind of good "reality check" in my opinion.

Application in Audit Assistant

Our Compliance Engagement standard "Tests of Controls" pages in the planning section start with the auditor describing the controls they wish to examine to see if they can be relied upon. Then there is a place to use some standard controls (such as "Evidence of authorisation") or create some others to test against the walk-through data. The sample items are chosen and added then the tests are applied, deviations are noted and results are evaluated as to what level these controls may be able to be relied upon. 

Then within the evidence gathering detailed substantive testing pages a sample is created to test the controls and to substantiate compliance with relevant regulations across the period. Deviations (if any) are noted and a decision is made as to whether further work is warranted. 

For example, from our walk-though tests we think that controls are good, so we decide to test 25 receipts over the year. We find that we have no deviations, so we are happy that controls over receipts are working and the regulations concerning receipts appear to have been complied with. 

If, at the other extreme, we are auditing a small Real Estate firm with no division of duties, we will not have any controls on which to rely, so we may decide to look at 60 receipts. However being a small entity they may only have issued 40 receipts for the year - so we will look at 100% which may all be covered by say 20 sales - and at the same time look at payments related to those deals as well. 

In these cases the CMA sampling option would not be appropriate - and correct response to the question that asks: "State sample size and selection method and rationale for this decision" would be "Other" - and the auditor would then say something like "based on preliminary assessment of controls over receipts as good, test 25 receipts over period randomly selected as per standard sampling policy". 

For Compliance jobs that have some quantitative element, we may want to use the CMA sampling tool option for the particular population we are testing. This would be applicable if we were say testing expense transactions in terms of compliance with the requirements of grant contract.