20130120 - Safety management systems in aviation maintenance (descriptive statistics) - 1997

[Data]

[<Normal page] [PEREZGONZALEZ Jose D (1997). Safety management systems in aviation maintenance (descriptive statistics). Knowledge (ISSN 2324-1624), 2013, pages 18-21.] [DOI]

Table of Contents

Generalization potential

AMO's SMS

McDonald et al analyzed the safety management systems of four aircraft maintenance organizations (AMOs)³ in 1997². This article presents newly re-worked descriptive results about that research.

Illustration 1 tabulates the main results so that each AMO (columns) gets ranked within each safety variable (rows). Rankings are in ascending order, with poor performance attracting lower rankings (eg, 1 or -4) and good performance attracting higher rankings (eg, 4 or -1). This procedure allows for adding up each organization's ranking at the bottom of the table to gain an appreciation of how good the organization is overall. The procedure also allows for setting an ordinal effect size of approximately 15 units on the resulting scale as a big effect size⁵.

Illustration 1 shows that there were sensible differences in safety management styles among the four organizations. Organization C, the best organization in the group, achieved position 43 on the scale (out of 50). At a sensible distance (15 units on the scale) was organization A in position 28, followed by organization B, also at a relatively sensible distance (12 units on the scale), in position 16. Organization D, the worst organization in the group, achieved position 10, only slightly lower (6 units on the scale) than organization D .

Illustration 1: Elements of safety management across four aircraft maintenance organizations
SMS elements	A	B	C	D
Safety policy	3, Good	2, Basic	4, Strong	1, Ineffective
Safety standards	2, Good	1, Basic	3, Strong	1, Basic
Planning & documentation	3, Good	2, Improving	4, Strong	1, Ineffective
Management training	2, Partly effective	1, Inconsistent	3, Effective	1, Ineffective
Auditing	1, Statutory	2, Basic	3, Strong	1, Statutory
Reporting	3, Backlogged	1, Starting	1, Starting	2, Poor
Incident investigation	3, Good	2, Ineffective	4, Strong	1, Absent
Feedback	3, Good	2, Ineffective	4, Strong	1, Poor
Change	3, Good	2, Ineffective	4, Strong	1, Absent

( SMS quality )	( 23 )	( 15 )	( 30 )	( 10 )

Climate & attitudes	A	B	C	D
Safety climate	3, Slightly positive	1, Negative	4, Positive	2, Slightly negative
Safety attitudes	2, Better	2, Better	2, Better	1, Good

( safety climate )	( 5 )	( 3 )	( 6 )	( 3 )

Disciplinary actions⁴	A	B	C	D
Impartial investigation	3, ≃53%	2, ≃35%	4, ≃62%	1, ≃28%
Incident discussed	3, ≃40%	1, ≃23%	4, ≃51%	2, =33%
Verbal reprimand	-2, ≃10%	-2, ≃10%	-3, ≃15%	-1, ≃7%
None	-3, ≃3%	-4, ≃4%	-1, ≃0%	-2, ≃3%
Disciplinary hearing	-4, ≃45%	-1, ≃23%	-2, ≃35%	-3, =37%
Suspension of license	-2, =15%	-4, =51%	-1, ≃8%	-3, ≃18%
Suspension of job	-4, =19%	-2, ≃9%	-1, ≃3%	-3, ≃13%
Disciplinary culture
( positive discipline )	( 6 )	( 3 )	( 8 )	( 3 )
( negative discipline )	( -15 )	( -13 )	( -8 )	( -12 )

Task procedures	A	B	C	D
Reported violations	-1, =30%	-2, =36%	-3, =37%	-4, =47%

( behavior )	( -1 )	( -2 )	( -3 )	( -4 )

SMS & culture (0-50)	28	16	43	10

Ordinal	small	medium	large	( Prototype diff. )
effect size⁵	5	10	15	( 16 )

Illustration 2 summarizes the correlations found between above groups of safety elements. In order to facilitate interpretation of the results, all ranks were transformed into positive values.

Illustration 2: Correlations (Spearman)
rho	SMS	Climate	+Discipline	-Discipline
Climate	.949
+Discipline	.949	1.000
-Discipline	-.200	-.316	-.316
-Behavior	-.400	-.211	-.211	-.800

As illustration 2 shows, there were high and positive ranked correlations⁶ between SMS, organizational climate, and positive discipline, which suggest that better safety management systems went together with better climate and more constructive disciplinary cultures (and viceversa) among these four organizations. Smaller and negative correlations appeared between above groups of variables and negative disciplinary expectations and violations, which also hint to relationships between better SMS, climates and constructive cultures and lower rates of negative disciplinary expectations and lower reported violations. Notwithstanding this, there was also a relatively high and negative correlation between preference for punitive disciplinary actions and the amount of reported violations.

Although little can be generalized from this particular research, above results describe a situation were safety management and safety culture did not appear standardized among similar organizations doing similar jobs and having similar responsibilities for aviation safety. Safety management and safety culture appeared as quite variable, with little cross-over across organizations, at least at the time of the research.

On the other hand, safety management and safety culture also appeared as quite consistent within each organization, further suggesting that the quality of the management system and the safety culture was idiosyncratic to each organization, was coherent within it, and permeated all levels in that organization.

Methods

Research approach

Exploratory study of the safety management systems of aircraft maintenance organizations³.

Design

Case study, where most information obtained during the research was compiled to provide a snapshot of the management style of four AMOs.

Sample

The ultimate sample was the four organizations, once all results were compiled according to provenance. Nonetheless, information for that sample was obtained by researching several populations and subsamples within each organization, each informing about different aspects of safety management at different organizational levels. The sample and subsamples were, thus, the following:

An overall sample of four AMOs, selected in a convenient manner among European organizations.
The population of four 'Maintenance Exposition Documents' (MOEs), the required formal documentation depicting maintenance roles and responsibilities, maintenance procedures and quality systems.
A sample of 33 maintenance and safety managers (including CEOs, production and quality managers, auditors and training personnel), selected in a convenient manner across all four organizations, who were interviewed about safety management.
622 participants, spanning from managers to technicians, selected in a convenient manner across all four organizations, who filled in a survey about safety attitudes and safety climate.
286 technicians selected in a convenient manner across all four organizations, who filled in a questionnaire about the use of official maintenance procedures in the job.

Materials

Content analysis of documentation.
Semi-structured interviews with management personnel.
Questionnaires, using a combination of dichotomous and Likert-like scales, as well as open-ended questions.

Overall, there is no much information about the validity and reliability of the different tools and procedures used. There are indications of face-validation for the results derived from content analysis as well as for sections of the surveys. The authors also went into some detail explaining how they adapted the safety and climate survey developed by Isla Diaz and Diaz Cabrera (1997¹). Overall, however, there is not much guarantee of proper validation or reliability assessment of the tools and procedures used.

Data analysis

Qualitative analysis (content analysis of documentation and interviews).
Quantitative analyses, namely frequency percentages and inferential analyses (ANOVAs and chi-square)⁷.

To above analyses carried out by the authors, the following analyses were introduced anew here:

Ranking of the AMO's according to perceived quality in all important variables.
Setting up of appropriate ordinal (unstandardized) effect sizes for interpretation purposes.

Generalization potential

This research has little generalization potential. The sample of AMOs is too small for carrying out inferential analyses, and little is known about the quality of the tools and procedures used. The convenient approach to sampling and tool selection, coupled with results which show rather idiosyncratic safety management styles within each organization, also limit generalization.

References

1. ISLA DIAZ Rosa & Dolores DIAZ CABRERA (1997). Safety climate and attitude as evaluation measures of organizational safety. Accident Analysis & Prevention (ISSN 0001-4575), 1997, volume 29, issue 5, pages 643–650.

2. McDONALD Nicholas, Siobhan CORRIGAN, Colin DALY & Sam CROMIE (1997). Safety management systems and safety culture in aircraft maintenance organisations. Safety Science (ISSN 0925-7535), 2000, volume 34, pages 151-176.

+++ Notes +++

3. Nowadays referred to as MROs, meaning 'maintenance and repair organizations', or 'maintenance, repair and overhaul organizations'.

4. Percentages of respondents who expected each particular disciplinary action from the organization as a response to an incident or accident. Percentages are mostly estimated from eye-balling a chart, as only a few of those were reported in the article.

5. For ranking purposes, a prototypical 'very poor SMS' can be defined as that with scores of 1 in all positive elements and scores of -4, or -3 or -2 when those are the top ranks in the variable, in all negative elements, which results in an added total of -10. Equally, a prototypical 'poor SMS' can be defined as that with scores of 2, or 1.5 when there are ties, in all positive elements and scores of -3, or -2 when there are ties, in all negative elements, which results in an added total of 6. (A similar procedure would also inform about prototypical 'good SMS' and 'very good SMS'.) The difference between adjacent prototypical SMS is about 16 units on the scale (15 units, for convenience). Given the multiplicity of variables and the complexity of safety management in aviation, moving up from one prototypical SMS to the next can be considered a big improvement (and viceversa). Thus, 15 units on the scale can be considered as a big effect size. In consequence, 10 units and 5 units can be considered as medium and small ordinal effect sizes, respectively. (The final scores are on a scale running between -10 and 40; however, in order to easy interpretation, this has been transformed into a positive scale running between 0 and 50.)

6. Given that the group comprised only four organizations, it was not possible to ascertain statistically the linearity of the correlations. Thus, a non-parametric approach was chosen for description purposes.

7. The original article did not provide information regarding the adequacy of doing such inferential analyses, though.

Want to know more?

FAA - Safety Management System: This FAA website provides more information about safety management systems in aviation.
Wiki of Science - Safety management systems in aviation maintenance (introduction): This Wiki of Science page is an accompanying introduction to above article.