Welcome to the General Aviation Advocacy Group of New Zealand

Client Satisfaction Surveys in a post-factual world…

The CAA is either living in a fools’ paradise and thinks all its customers are happy, or it is deeply afraid to ask them. We can’t think of any other reason why the CAA refuses to hire a neutral organisation and tell it go to the customers and ask some honest questions. We also have convincing proof that no company or organisation can rely on anything other than an independent monitoring of its customers.

Here’s why…

In January this year, we wrote to the Chairman of the CAA Board pointing out that the Authority engaged Colmar Brunton to undertake a client satisfaction survey in 1995 and 1998; five years later in 2003, the same research company carried out a similar survey. Since then, there have been no further CAA Client Satisfaction Surveys specifically targeting all CAA document holders. The 2003 survey’s results did not heap praise upon the CAA.

In his reply, the Chairman said:

“Your interest in surveys is topical as the CAA will shortly be repeating its flagship ‘Feel Safe’ survey conducted by Colmar Brunton. Management are currently planning on supplementing the survey, or even splitting its content, so that while retaining its prior focus on the CAA’s ultimate customer (the public) it also provides more information from the aviation sector”.

OK, it’s costing you $4.50 a minute. But do have a problem with me?

The Colmar Brunton “Feel Safe” surveys have typically only contained results from 300 to 400 respondents. We have written to the Chairman and asked that all CAA document holders be made aware of his planned consumer survey. But we reminded Mr Gould that this is just a consumer survey, and what is actually required is a CAA document-holder survey related to specific client issues, and promoted via the CAA’s email user database and Vector magazine.

Post-audit or inspection feedback to the CAA

In our letter to the Chairman, we pointed out that MoBIE has for many years operated a system in which, after every investigation, an inspector is required to send a client satisfaction survey to all parties involved. Something similar would be a valuable tool for the CAA Director for feedback about, to quote just one example, the knowledge and performance of auditors and the way they interact with aircraft operators and aircraft maintenance facilities. A client feedback system also provides useful information to senior managers of departments to gauge the performance of individual staff members as part of their yearly performance reviews.

The Chairman said:

“… at the end of each audit or inspection, the CAA already surveys the parties involved in order to gather information on their views on the effectiveness and efficiency of the regulatory intervention they have just been involved in. While that provides limited information about the performance of one regulatory function, we are interested in gathering a broader range of views to inform risk management and decision-making. Both the management and Board of the Authority recognize the importance of such information from the sector to inform our planning and improvement efforts. In part, this is one of the reasons we place so much importance on the relationship with representative bodies like the Aviation Community Advisory Group.”

We then sought to verify what the Chairman had claimed was correct – “the CAA already surveys the parties involved in order to gather information on their views on the effectiveness and efficiency of the regulatory intervention they have just been involved in”.

We asked a number of operators if what the Chairman claimed (or had been told by his officials) actually happened in practice.

  1. Did the Chairman’s statement have validity?
  1. Did operators feel comfortable providing feedback to the CAA when their AOC is at stake?

We believe the replies below speak for themselves, and we’d welcome further comments.

  • No, I’m not aware of any statement like that. Imagine if an operator criticised them at that point, they could be deemed not fit or proper and their AOC withdrawn or delayed when the auditors got back to Wellington. Yes there is a discussion, but who would dare to complain? It proves how out of touch the Chairman is.
  • I never had the auditors ask me how the audit went or what I thought of CAA. If you wanted a good audit report you would never consider expressing your thoughts on that issue. Ask the Chairman for some of the recorded comments.
  • No, I haven’t been asked for feedback (that’s not to say I haven’t given verbal feedback to auditors at a subsequent audit however!).
  • I am not aware of any formal feedback system. At one point during an audit, I found I was being asked the same questions at each audit such as “Is your fax number still xxxxxxxxx. Is your mailing address still xxxxxxxxx. Is your phone number still xxxxxxxx. When I’m paying over $4.50 PER MINUTE for CAA to ask me the same questions at each audit to which there was no change, I wasn’t happy. I suggested they send me an A4 sheet with those questions on it prior to the audit and I could go through and tick the boxes, thus saving valuable minutes. They thought that was a great idea but nothing was done.
  • I have never been asked at all, fair to say I have offered my opinion on a number of occasions regarding their efficiency levels, no one has ever asked me for it tho and I have undergone approx 25 audits over the years.
  • The last CAA audit that I went through was about 2 years ago for our former Part 145 maintenance organisation. There was certainly a discussion on how we viewed the conduct of the audit but our responses were very much along the lines of what you would expect of “the regulated” talking to “the regulator”. No matter how the CAA Chairman might like to dress it up, it is extremely difficult to get anyone whose livelihood relies on holding an aviation document to be critical (no matter how well intentioned or constructive) of the organisation that issues said document.
  • I am in full agreement for the auditing process to be outsourced, what a great idea, at least if this happened there would be no favoritisms by the auditors. It’s imperative that all operators record* audits.  In that way it keeps the auditors honest and semi under control.
  • Yes this is an interesting subject and I totally agree with the other comments.  After last year’s audit we received a survey, which I filled out with my own feelings regarding CAA.  But I then thought it wasn’t a good idea to return it as it may have been held against me.  So I never posted it. I am certain that industry is running scared of CAA and feel that there are “bully” tactics going on. CAA is determined to change the aviation industry to make it highly regulated to cover their own arses.  Hence industry is finding this hard to come to terms with.
  • A further point about commenting on audit skills etc is as long as they are sitting at your table you are paying for that time.  You want them in and out of your place as soon as possible so the audit doesn’t cost you an absolute fortune.  So no, we wouldn’t think there would be anyone out there brave enough to criticise the CAA openly and in front of the auditors, as it would make their lives a misery.

The Chairman also said:

“On the information theme, please note that management is also introducing a ‘balanced scorecard’ to improve the information available on organisational performance. While not yet complete, over the next 12 months you can expect to see information being sought from the sector and others to inform the assessment of matters of regulatory effectiveness, efficiency and regulatory burden”.

  • Record, in this context, means to “tape record”. The GAA recommends that all important dialogue you have with CAA personnel –  on your premises, over the telephone or at the CAA’s office – should be recorded on audio and/or video. Before making any recording, you should advise your intentions to the CAA personnel, ensure they agree to an uncensored recording and offer to provide them with a copy of it.

Speak Your Mind