Evaluation of training using different modules.
Bolstad et al1 carried out the research on evaluation of different computer based situation awareness training for the general aviation (GA) pilots. Results (see illustration 1) shows how effective basic skills training can be for the pilots with low flying hours. Checklist questionnaires, ATC comprehension and psychomotor skills were tested for each group of pilots. There was a drop of mean in post test performance (-.20) in psychomotor skills task ( tracking error) in the control group. However, the control group post test mean results were always better than their mean pre-test results in the checklist completion questionnaire, SAGAT queries on winds and altitude. This suggest that even the control group who had limited basic training, seemed to improve from their pre-results. Mean for the control group were always lower. This may be due to the fact that the control group was assigned the tedious task of reading and assembling 3-D figures for seven hours. It could be also due to their currency where the control group had an average of 3 months since their last flight compared to the experimental group with just 10.4 hours since their last flight. As a result the experimental group were more current and were up to date with the ATC comprehension, checklist completion and psychomotor skills. Training modules selected for the control group were also not directly associated with flying. In contrast, the experimental group was given more relevanct tasks that were directly associated with flying. For example, module was designed to enhance SA in GA pilots. A joy stick which is similar to control columns used by pilots in flying to gain their flying experience was used to train for the improvement of psychomotor skills, and a Microsoft visual basic module was provided as part of comprehension training.
Compared to the control group, the experimental group always outperformed in checklist completion questionnaire, psychomotor skills task at highest difficulty level, SAGAT query on current winds as well as altitude. This shows that training modules may have the potential for improving basic skills in pilots with low hours. Results showed that out of 15 SAGAT queries, the significant difference in change in score in experimental and control group were found in just two queries of current altitude and wind. This suggest out that majority of queries asked had no significant between two groups.
| Illustration 1. Mean difference of pre and post-training performance by group | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Results analysed by | Variable measured | Me | Mc | Result | Better performance by | |||||||
| Basic skill performance | Checklist completion questionnaire | 3.33 | .42 | significant | experimental | |||||||
| Reaction time | — | — | not significant | — | ||||||||
| No. of correct responses | — | — | not significant | — | ||||||||
| psychomotor skills task | Highest difficulty level | .66 | -.20 | significant | experimental | |||||||
| Tracking error | — | — | not significant | — | ||||||||
| SA accuracy | SAGAT queries | |||||||||||
| current winds | .33 | .19 | significant | experimental | ||||||||
| current altitude | .29 | .01 | significant | experimental | ||||||||
| on time | — | — | not significant | — | ||||||||
| current pitch | — | — | not significant | — | ||||||||
| indicated fuel | — | — | not significant | — | ||||||||
| altimeter setting | — | — | not significant | — | ||||||||
| current heading | — | — | not significant | — | ||||||||
| indicated airspeed | — | — | not significant | — | ||||||||
| ATC control in contact | — | — | not significant | — | ||||||||
| current altitude vs. planned | — | — | not significant | — | ||||||||
| conformance with clearance | — | — | not significant | — | ||||||||
| distance of special airspace | — | — | not significant | — | ||||||||
| planned vs current ground speed | — | — | not significant | — | ||||||||
| change required avoiding obstacles | — | — | not significant | — | ||||||||
| change required avoiding certain airspace | — | — | not significant | — | ||||||||
| Flight skills performance | — | — | not significant | — | ||||||||
| Me= mean of experiment group; Mc= mean of the control group; SAGAT= Situational Awareness Global Assessment Technique | ||||||||||||
Illustration two points out that the most progress in pre-test and post-test results of different variables by ranking order from one (best) to eight (worst). Illustration also shows that exp. group checklist completion questionnaire performed excellent whereas, exp. group psychomotor skills (highest level difficulty) and control group checklist completion questionnaire performed better. It is also noted that exp. group checklist completion questionnaire, exp. group SAGAT query (current altitude) and control group SAGAT query (current wind) performed good where as control group of SAGAT query (current altitude and Psychomotor skills ( highest level difficulty) performed bad.
| Illustration 2. Performance of variable by ranking highest to lowest mean | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Variable | Group | Mean difference pre and post | Rank | Performance interpretation * | ||||||||
| Checklist completion questionnaire | experimental | 3.33 | 1 | excellent | ||||||||
| Psychomotor skills ( highest level difficulty) | experimental | 0.66 | 2 | better | ||||||||
| Checklist completion questionnaire | control | 0.42 | 3 | better | ||||||||
| SAGAT query: current wind | experimental | 0.33 | 4 | good | ||||||||
| SAGAT query: current altitude | experimental | 0.29 | 5 | good | ||||||||
| SAGAT query: current wind | control | 0.19 | 6 | good | ||||||||
| SAGAT query: current altitude | control | 0.01 | 7 | bad | ||||||||
| Psychomotor skills ( highest level difficulty) | control | -0.2 | 8 | bad | ||||||||
| *mean difference of < 1 is excellent, mean difference of 0.36 to .66 is better; mean difference of .02 to .35 is good; mean of 0.01 to -.34 is bad. | ||||||||||||
Methods
Research approach
- This was an exploratory research to examine how effective basic skills training is for pilots with low flying hours using three modules.
Sample
- A convenience sample of 24 pilots from the Embry-Riddle Aeronautical University(ERAU) in Daytona Beach, Florida. The experimental group has 12 pilots with a average flight hours of 98.4 and the control group has 12 pilots with average of 104.3 flight hours.
- 2 participants in each group had instrument rating .
- Pilots were chosen from ERAU which represented typical GA pilots. Although important to generalise to populaton about hte method was important to mention, how participants were self selected or randomly selected from ERAU is not given.
Design:
Between group design. Participants were not assigned to each groups randomly and this does not ensures extraneous variable are equally split. Thus, this will help in becoming confounding variable.
Variable
- Dependent Variable: Checklist completion knowledge, ATC comprehension, psychomotor task performance, SA accuracy and flight skills performance.
- Independent Variable: Three SA training modules.
- Confounding Variable: The experimental group had average of 10.4 hours since last flight and control group had average of three months since last flight. This lack of currency of 3 months of control group may have had an negative effect on their performance.
Materials
- Performance of 5-item questionnaires to asses pilot's knowledge of checklist procedure, psychomotor skills training ( highest level of difficulty, tracking error) and ATC comprehension training module (response and reaction time) were collected pre and post training.
- Macromedia director was used to describe the problem that can occur when checklist is not completed, reasons why they are not completed and methods to guide pilots in carrying out checklist.
- Microsoft visual basic module to check ATC comprehension training.
- Vigilance was assessed using the Mackworth clock test taken from Teikari (1977) which was run in a computerised environment.
- Aeronautical Information Manual/ Federal Aviation Regulation 2000, The pilot's radio communication handbook and Interlocking Kinex brand constructions sticks
- Microsoft Flight Sim 2000, 15 SAGAT queries, personal computer with three screens for SA training module, headphones, joystick and yoke.
Procedure
- Participants were asked to complete the checklist questionnaires and they performed pre-training ATC comprehension, psychomotor skills task and flight scenario. Before flight, practice flights were given to the pilots to practice on SAGAT and the simulator.
- In their own time, participants of both groups were told to spend 7 hours completing training modules. Each pilot kept their log recording of the time spent on training and software that is used by experimental group recorded the participants time on task for comprehension training and checklist completion training.
- After completion of the training, participants completed the post-training measures for basic skills performance and post training flight scenario.
- Finally, participants were debriefed and paid nominal fee for their participants.
Generalisation potential
- As sample was taken from large university with about 2000 student pilots, these results could be applied to typical GA pilots of similar age with similar flying hours of 98 to 104 .
- As convenience sample was taken without any randomisation of selection process, the results should be generalised to similar university population only.
Want to know more?
Situation Awareness : This pages defines SA, level of SA, factor influencing and challenges of SA
Original article : Full original article of this research can be obtained here.
Contributors to this page
Authors / Editors
Abilash Thapa (2013), Massey University, New Zealand (Thapa Abilash)
