Generating Audit Findings and Conclusions
by J.P. Russell
The ability to identify audit findings, communicate them and determine the audit conclusions is one of the skills that adds the most value to a management system audit.
ANSI/ISO/ASQ QE19011S-2004, Guidelines for Quality and/or Environ-mental Management Systems Auditing (ISO 19011)1 is a valuable auditing resource. But few actually will read or study the guideline standard because of its cost, rather dry content or an inability to see its relevancy.
That lack of use is unfortunate because ISO 19011 is the result of input from knowledgeable practitioners in the field of auditing all over the world. As an introduction to this resource, I will review the content of two of its most important clauses: 6.5.5, generating audit findings, and 6.5.6, preparing audit conclusions.
Generating Audit Findings
After an audit team collects the facts and completes its investigation, it is time to determine the results of the investigation. For audits, the results are called reported audit findings.
The first step is to evaluate the evidence against the audit criteria. The evidence is the factual information collected or observed during the performance of the audit. The audit criteria are the standards, procedures, regulations or objectives the organization was audited against. The criteria represent requirements the organization must comply with.
ISO 19011 says the audit findings can indicate conformance or nonconformance with the audit criteria. Some audit programs require auditors to report evidence of conformance as well as evidence of nonconformance, but most audit reports contain only the facts that support a nonconformity or noncompliance.
If one of the audit objectives is to identify
opportunities for improvement, the findings might include
observations of inefficiency or ineffectiveness (see
“generating audit findings” in Figure 1). If there is
an audit team, it should have met at various stages of the audit
to review audit findings or potential findings.
Conformity to requirements (audit criteria) should be summarized. The summary should indicate locations, functions or processes that were audited. This already should be in the individual audit scope, but perhaps more specifics will be needed for the findings.
For example, if the filing department (function) of the Chicago office (location) of the organization being audited is noted to be in conformance with records control requirements, this is an example of how audit findings can be positive as well as negative.
If included in the audit plan, evidence of conformity also must be recorded and presented as audit findings. Evidence of conformity might be necessary for high risk processes or if legal requirements are part of the audit criteria.
When Findings Are Not OK
For most audits, only evidence of nonconformity is recorded. Auditee organizations tend to want to know what is wrong and what needs to be fixed rather than what is OK and needs no action. Nonconformities and their supporting audit evidence should be recorded. The audit report could be the record of nonconformities.
The guidance standard further says nonconformities might be graded. From this we also can conclude they might not be graded. Historically, nonconformities have been graded as major or minor, but some audit organizations simply report nonconformities, believing the auditee is the best judge of the significance of the nonconformity.
This might be true because the auditee organization knows its process better than a third-party or even a first-party auditor. On the other hand, auditors best know the significance of nonconformities relative to the standard or audit criteria. Plus, a decision not to grade might cause auditors to be lazy and collect only evidence of imperfection rather than enough evidence to identify systemic issues.
The auditee should review the findings (see “auditee reviews findings” in Figure 1). The lead auditor should seek acknowledgement from the auditee that the evidence is accurate and that the auditee organization understands the nonconformity or noncompliance. In many cases the auditee initials the nonconformity statements, or there is a statement at the exit meeting that the nonconformities were reviewed and acknowledged.
This section of ISO 19011 ends with a statement that every attempt should be made to resolve any diverging opinions concerning the audit evidence or finding.
If there are diverging opinions, an auditor can review the supporting evidence and ask for feedback about its accuracy. He or she also can ask for new evidence that would contradict the existing evidence or support a different finding.
Resolving divergent opinions supports an evidence based (let the facts speak for themselves) approach. If the evidence collected is wrong, it should be corrected. If the evidence is accurate, the findings should stand. You cannot always get agreement, so any unresolved issues should be recorded.
Audit findings are not always nonconformity statements. For internal audits, a nonconformity might be put directly on a corrective action request form instead of on a nonconformity form.
Some internal audit program procedures might skip the generation of separate nonconformity and corrective action request forms. If the audit objective is to determine project implementation status or gaps, findings might be related to project progress instead of nonconformities. For example, a supplier might be implementing new controls to reduce or eliminate customer appraisal costs.
Preparing Audit Conclusions
Audit findings or nonconformities might be generated throughout the audit, but audit conclusions can be determined only at the end of the investigation.
For audits taking one day or less, generating of audit findings and conclusions might take place at the same review meeting. For external audits, the review meeting normally takes place immediately at the end of the data gathering phase.
For internal audits, the review meeting could be scheduled at a later date to accommodate organizational needs. However, sooner is better so individual auditors still can recall or decipher notes clearly from situations encountered during the audit.
Auditors should review findings and any other information relevant to the audit objectives. Examples include:
- Two areas still must be audited before certification or license can be granted.
- The organization’s only certified technician is retiring at the end of the week with no replacement identified.
- Factual information is needed to qualify or quantify a particular audit conclusion.
Reviewing findings and other relevant information brings the audit full circle—when outputs are compared to input requirements.
ISO 19011 says the audit team should agree on the audit conclusion, taking into account the uncertainty inherent in the audit process. In most of the audits I have taken part in, the audit team leader determines the audit conclusion and seeks consensus from the audit team.
ISO 19011 says audit team leaders should go an extra step and secure agreement from each individual auditor on the audit team.
If specified in the audit plan or audit objectives, recommendations should be prepared. Some believe auditors should not make recommendations because the auditee then will do what the auditor recommends without considering more optimal solutions. Others believe that if the auditor has a solution, he or she should share it so the problem can be fixed as soon as possible.
Because the word “recommendation” is not defined or explained in the auditing guideline standard, this is still a very fuzzy area. Typically, recommendations are not made because the integrity of the audit process could be compromised. On a subsequent audit, the same audit organization or auditor could be verifying its own corrective action recommendations, resulting in a conflict of interest.
For second-party audits (audits of suppliers by the purchasing company), a recommendation could be misinterpreted as binding or a contract requirement.
My experience is that making recommendations to address findings is problematic and detracts from the value of the audit.
It is also necessary to discuss audit follow-up activities if there is a nonconformity or noncompliance. For a third-party audit, the follow-up might be the responsibility of another group or department and might not involve auditors. For internal audits, the same auditor or an auditor from the same audit program department might conduct a follow-up audit to verify the nonconformity was corrected.
The audit plan should indicate the follow-up action expectations, and audit program procedures should be followed.
ISO 19011 also contains practical help for some clauses—an interesting feature you will not see in many standards.
Audit conclusions can address several issues (see “preparing audit conclusions” in Figure 1). The practical help section lists three issues that conclusions can address:
- Audit conclusions can estimate the extent of conformity of the management system against the audit criteria. This is typical of most conformity audits.
- Audit conclusions can include a statement about the effective implementation, maintenance and improvement of the management system. During an audit, the audit team members will observe how the management system was deployed and its effectiveness. It also will observe whether the system is being properly maintained based on adhering to requirements, correcting nonconformities and taking corrective action. Improvement might be realized through preventive and innovative actions.
- Audit conclusions can assess the capability of the management review process. The audit evidence might support a conclusion that management has ensured the continuing suitability, adequacy, effectiveness and improvement of the management system.
The practical help section ends with a statement about recommendations. If specified in the audit objectives, audit conclusions might lead to recommendations regarding improvements, business relationships, certification/registration or future audit activities.
For example, the audit team might recommend certification/registration of the management system or that oversight be reduced due to the maturity of the management system. For a supplier audit, the audit team could recommend acceptance of the organization to the highest supplier qualification level. My experience is that making recommendations based on conclusions should be encouraged and might add value.
Importance of Fundamentals
The U.S. version of ISO 19011 has supplemental guidance for first-party, second-party and small organizations. The international version is heavily slanted toward third-party audits. In this article, I’ve tried to incorporate some of that supplemental guidance along with some of my own examples.
Audit program procedures of individual organizations might differ from the ISO 19011 guidance due to business situations and audit program objectives, but we need to stay in touch with the fundamental guidance provided by our experts to ensure we are on solid ground.
- ANSI/ISO/ASQ QE19011S-2004, Guidelines for Quality and/or Environmental Management Systems Auditing (ISO 19011), ASQ Quality Press, 2004.
J.P. RUSSELL is president of J.P. Russell & Associates, Lake Wylie, SC, and managing director for Quality WBT Center for Education at www.qualitywbt.com. He is a fellow of ASQ, voting member of the American National Standards Institute/ASQ Z1 committee and member of the U.S. technical advisory group for International Organization for Standardization (ISO) technical committee 176. Russell is an ASQ certified quality auditor and author of several Quality Press books, including Process Auditing Techniques, Internal Auditing Basics and the ISO Lesson Guide 2000.