Hours, Experience and Competence in Commercial Aviation

Todd and Thomas undertook an exploratory study to determine whether the industry standard of cumulative hours showed any relationship with the technical performance of commercial pilots[1]. They were interested in the common perception that hours equals experience equals competence, and therefore that those with fewer hours are less competent than those with more hours. This is an issue for an industry that is facing a global pilot shortage combined with new multi-crew licences requiring fewer total hours and students who learn in condensed academy courses rather than the traditional "work your way up" style of many different jobs. The authors held the hypothesis that there would be a difference in technical skill based on flight hours.

Todd and Thomas sampled pilots from a single commercial airline running high-capacity short-haul operations. The sample of pilots was split into four categories – high and low hour Captains, and high and low hour First Officers (F/Os). The categorisation of the sample is shown in Illustration 1. The way the sample was obtained gave the group a very even split in terms of rank (Captains vs F/Os) with the most common participants being high-hour Captains.

Illustration 1: Sample categories by percentage
Category by hours	Percentage
F/O Low-hour (<1500 hours)	7.4%
F/O High-hour (>1500 hours)	36.1%
Captain Low-hour (<5000 hours)	8.4%
Captain High-hour (>5000 hours)	48.1%

N = 285

No statistically significant differences were found between or within the hour categories in technical performance. Most airlines have standard operating procedures which are used for both normal operations (including stabilised approaches) and as assessment criteria and these are obvious in the results. Illustration 2 shows the stabilised approach criteria studied, and the percentage of achievement for each rank (captains and F/Os). The data shows similar achievement for all criteria, with lower achievement in the area of being “on the flight path” for ranks. This could be due to pilots hand-flying certain types of approach due to instructions from air traffic control, something which could be expected to reduce in frequency as next-generation navigation procedures are implemented. The data collected did not provide information as to whether pilots were below the flight path or above it. In terms of instrument flight conditions (flying in cloud) and overall safety, being above the flight path could result in a missed approach which is inconvenient and has fuel and schedule implications, but is safer than being below the flight path which, in absence of warnings and immediate actions, could result in an undershoot and crash prior to reaching the runway.

Illustration 2: Stabilised approach criteria achieved at 500ft by pilot rank²
Stabilised Approach Criteria	Captains	F/Os
Airspeed	98.1%	96.0%
Vertical Speed	96.9%	96.8%
Appropriate Thrust	96.9%	96.8%
Landing Configuration	98.8%	97.6%
On Flight Path	92.5%	88.7%
Briefings and Checks	99.4%	99.2%

Overall, the common perception that more flight hours are equal to greater technical competence appears to be unfounded. F/Os showed no significant difference in their abilities than Captains despite have (in some cases) substantially lower hours. Given that F/Os with over 1500 hours could possibly have the same or more hours than Captains with less than 5000 hours, Illustration 3 contrasts F/Os with low hours and Captains with high hours to further investigate the possibility of experience leading to differences in technical performance. The data shows that the F/Os <1500 hours achieved the stabilised approach criteria 100% of the time, whereas the the high-hour Captains did not. This is likely caused by the difference in autopilot disconnect height between the two groups. The low-hour F/O's remained flying by autopilot to a lower level which gave less time for their manual flying ability to interfere with the approach set-up. Captains with >5000 hours disconnected at almost 1200 ft which meant they were hand-flying the approach for longer, resulting in slightly lower achievement in the criteria.

Illustration 3: Contrasting the extremes
	F/Os <1500 hours	Captains >5000 hours
Height at Autopilot Disconnect	655.0 ft	1198.0 ft

Stabilised Approach Criteria at 500 ft²
Airspeed	100%	97.7%
Vertical Speed	100%	96.2%
Appropriate Thrust	100%	96.2%
Landing Configuration	98.5%	96.9%
On Flight Path	100%	92%
Briefings and Checks	100%	99.2%

Given the industry's continual move to more highly technical and automated aircraft, navigation systems and air traffic management systems, it is likely that low-hour pilots who have been trained to a standard level of competence will be just as able to safely pilot commercial aircraft as those who meet historical (and current) hour-based experience levels.

Methods

Research approach

The authors conducted this exploratory study to examine whether cumulative flight hours had any effect on technical pilot performance in a commercial environment.

Sample

The sample comprised was comprised of 302 observations of flight sectors from one short-haul airline within Australia, of which 287 observations were found to suitable for analysis. The sectors that were observed were chosen to give a stratified sample which best represented the airline’s route structure. The sampling took place over a six week period and was collected during routine line operations safety audits (LOSA). Due to the anonymous nature of the observations, individual pilots may have been observed more than once.

Design

The study grouped the pilots into four groups, and was a between-groups factorial design to investigate the relationship between flight experience (hours) and technical performance. Technical performance was assessed for the pilot flying against set stabilised approach criteria at both 1000 feet above airfield elevation (AFE) and 500ft AFE, as well as the height the pilot disconnected the autopilot. This analysis uses only the 500ft and autopilot disconnect data, as well as sample population information.

Variables

Independent Variable

Non-manipulated - The experience group of the pilot

Dependent Variables

Categorical - The performance against the criteria for a stabilised approach
Continuous - Autopilot disconnect height

Procedure

The observations were embedded as part of a normal line operations safety audit (LOSA). Neither the observers nor the pilots were aware of the hypothesis regarding flight hours and performance. The observers had attended a training course and had established an interrater reliability between 0.73-1.0, with anything greater than 0.70 being acceptable. The pilots were marked on whether or not they met the established criteria on each approach, and at what height they disconnected the autopilot.

Data analysis

The original article used Chi-square analyses for the categorical variables ie each one of the stabilised approach criteria, and one-way ANOVA for the continuous variable of autopilot disconnect height.
This analysis uses descriptive statistics to further contrast the differences between the pilot groups.

Generalization potential

Given the regulated standard required for licence and rating acquisition, and the industry-wide practice of hiring based on hours accumulated, this study could be generalised to the wider commercial pilot population operating in similar conditions.

References

1. TODD Melanie A & Matthew J THOMAS (2012). Flight hours and flight crew performance in commercial aviation. Aviation, Space and Environmental Medicine, August 2012, volume 83, number 8, pages 776-783.

+++ Footnotes +++

2. Percentage values for the criteria refer to the proportion of flight sectors where the criteria were met.