SFTY 320 Embry Riddle Aeronautical University Simulator Fidelity Training Paper

  

Refer to the article attached titled Flight Simulator Fidelity, Training Transfer, and Role of the Instructors in Optimizing Learning and other attached readings and these videos,

Describe the meaning of fidelity, training transfer, and negative training transfer and do the following.

With respect to flight simulators, examine fidelity and whether it changes over time or when learning progresses.

Explore the instructor’s role and responsibilities in the training and simulation environment.

Finally, employ some best practices to determine fidelity shortfalls for specific simulator tasks by individuals.

This assignment is supposed to be the majority construct of the learner, so please don’t use too many references. It should be written in a blog style.

665200
HFSXXX10.1177/0018720816665200Human FactorsTeam Training
Team Training for Dynamic Cross-Functional
Teams in Aviation: Behavioral, Cognitive, and
Performance Outcomes
Glenn E. Littlepage, Michael B. Hein, Richard G. Moffett III,
Paul A. Craig, and Andrea M. Georgiou, Middle Tennessee
State University, Murfreesboro
Objective: This study evaluates the effectiveness of
a training program designed to improve cross-functional
coordination in airline operations.
Background: Teamwork across professional specializations is essential for safe and efficient airline operations, but
aviation education primarily emphasizes positional knowledge
and skill. Although crew resource management training is
commonly used to provide some degree of teamwork training, it is generally focused on specific specializations, and little
training is provided in coordination across specializations.
Method: The current study describes and evaluates a
multifaceted training program designed to enhance teamwork and team performance of cross-functional teams
within a simulated airline flight operations center. The
training included a variety of components: orientation
training, position-specific declarative knowledge training,
position-specific procedural knowledge training, a series of
high-fidelity team simulations, and a series of after-action
reviews.
Results: Following training, participants demonstrated
more effective teamwork, development of transactive memory, and more effective team performance.
Conclusion: Multifaceted team training that incorporates positional training and team interaction in complex
realistic situations and followed by after-action reviews
can facilitate teamwork and team performance.
Application: Team training programs, such as the one
described here, have potential to improve the training of
aviation professionals. These techniques can be applied to
other contexts where multidisciplinary teams and multiteam systems work to perform highly interdependent
activities.
Keywords: training evaluation, team coordination, simulation and training, multiteam systems, group performance
Address correspondence to Glenn E. Littlepage, Psychology
Department, Middle Tennessee State University, Murfreesboro,
TN 37132; e-mail: Glenn.Littlepage@mtsu.edu.
HUMAN FACTORS
Vol. 58, No. 8, December 2016, pp. 1275­–1288
DOI: 10.1177/0018720816665200
Copyright © 2016, Human Factors and Ergonomics Society.
Introduction
The objective of this study is to describe and
evaluate the effectiveness of a theoretically and
empirically based program of team training for
entry-level aviation professionals. The multifaceted program is designed to facilitate teamwork,
emergent cognitive states supporting teamwork,
and team performance. Our research and training
program emphasizes the importance of teamwork
across all the positions within a multiteam environment and provides participants the opportunity
to refine teamwork skills in a nonconsequential
environment. Specifically, we examine a training
program to improve the functioning of a simulated airline flight operations center.
Teamwork in Aviation
Work teams are an integral component of organizations (Devine, Clayton, Phillips, Dunford,
& Melner, 1999; Lawler, Mohrman, & Benson,
2001), and the commercial aviation industry is
no exception (Kanki, Helmreich, & Anca, 2010).
A number of highly trained professionals with
complementary expertise are needed to operate
an airline. Work teams operate on the flight deck,
in the cabin, at air traffic control centers, at maintenance facilities, and at airline flight operations
centers. Not only must these individuals be proficient in their respective specializations, they must
also work together in a coordinated manner. Poor
coordination across occupational specializations
may be problematic (Beer, 2013), especially when
nonroutine events occur (Bienefeld & Grote,
2014; DeChurch & Marks, 2006; Marks, Mathieu,
& Zaccaro, 2001; Salas, Sims, & Burke, 2005).
Misunderstandings and poor coordination can
lead to costly flight delays (Ball et al., 2010) and
greatly compromise safety (Merket, Bergondy, &
Salas, 2000; Shappell et al., 2007).
1276
Recognition of the importance of effective
teamwork led to the development of crew resource
management (CRM) programs designed to
improve crew coordination and decision making
(Federal Aviation Administration, 2004; Helmreich, Merritt, & Wilhelm, 1999). CRM training is
a common and effective approach to train teamwork knowledge and skills (O’Connor et al., 2008;
Salas, Wilson, Burke, & Wightman, 2003). However, almost all CRM and related teamwork training has been done with teams consisting of members who share a single occupational specialization (i.e., pilots, cabin crew, air traffic control, or
maintenance; Kanki et al., 2010).
Although cross-specialization coordination is
essential to airline operations, little research has
been directed toward the study of coordination
within cross-functional teams or within multiteam
systems in aviation. One exception is a study of
coordination between cockpit crews and flight
attendants (Bienefeld & Grote, 2014), which
showed that coordination between the captain and
the lead flight attendant led to more effective multiteam performance. These authors and others
(e.g., Shuffler, Jimenez-Rodriquez, & Kramer,
2015) suggest that team training programs might
improve teamwork across occupational specializations and multiteam performance.
One important component of commercial aviation that requires teamwork across occupational
specializations is the flight operation center. Each
airline maintains a centralized flight operations
center, which is the operational management center for the airline’s flights. Within this center,
members of diverse aviation specializations work
in a coordinated manner to perform the activities
needed to ensure that flights are staffed, loaded,
fueled, and launched in a way that promotes safe
and efficient airline operations. Because of the
requirement for intensive coordination between
individuals with different domains of specialized
knowledge, the flight operations center represents
a cross-functional team. Not only must members
of a flight operations center work with other disciplines at the center, but they must also work
closely with cockpit crews in a number of planes
and with personnel at various airports served by
the airline. Thus, these members operate within a
multiteam system. A multiteam system involves
multiple-component teams (e.g., a flight crew,
December 2016 – Human Factors
flight operations center personnel, and airport
ramp and gate personnel) that work interdependently to accomplish shared goals (Mathieu,
Marks, & Zaccaro, 2001; Zaccaro, Marks, &
DeChurch, 2012). Effective multiteam performance requires coordination both between team
members and also between component teams.
Although aviation professionals receive extensive education and professional training within
their specialization, they frequently lack training
for coordination with other specializations. Thus,
they are generally less than optimally prepared to
work in the team and multiteam settings that are
common in the airline industry. Collaboration
across occupational specializations is difficult
because of differences in task orientation dimensions, such as speed versus accuracy and analysis
versus action (Caruso & Woolley, 2008). Consequently, team training may be especially critical in
flight operations centers because people with different functional backgrounds must work together
both within the center and also in dealings with
outside individuals or groups. Theory and research
on group processes, group/team performance, and
multiteam systems (DeChurch & Marks, 2006;
Zaccaro et al., 2012) provide perspectives from
which to view the cognitive states and behavioral
processes that facilitate the level of coordination
required to maintain efficient airline performance.
Team Training Theory
A variety of sound training approaches have
been identified (Gregory, Feitosa, Driskell,
Salas, & Vessey, 2013) and the current program
incorporated multiple practices. Consistent with
the training framework suggested by CannonBowers and colleagues (Cannon-Bowers, Rhodenizer, Salas, & Bowers, 1998), the training incorporated prepractice, practice, and after-practice activities. Prepractice activities included onboarding,
position-specific declarative knowledge training, and position-specific procedural knowledge
training. Practice activities involved a series of
multiperson dynamic simulations. Postpractice
activities involved after-action review sessions
following each simulation.
Onboarding (orientation training) was
designed to achieve two important functions. It
served to provide motivation and orientation.
Team Training
Effective onboarding can have a big influence
on employee motivation (Snell, 2006) and can
promote the development of shared and accurate
mental models (Marks, Zaccaro, & Mathieu,
2000).
Position-specific training can provide declarative knowledge (facts and concepts) and procedural knowledge (an understanding of how to
accomplish the task). Both are needed for effective team performance (Banks & Millward, 2007).
The purpose of the position-specific training was
to ensure that each participant developed an understanding of his or her role, responsibilities, and the
technical knowledge to do the job.
Multiperson simulation that captures the essential task and teamwork functions is considered to
be an effective approach to team training (Howard, 2011; Salas, Bowers, & Rhodenizer, 1998;
Salas, Cooke, & Gorman, 2010). A recent metaanalysis of team training showed simulation-based
training was correlated (r = .45) with team outcomes (Klein et al., 2005, cited in Kozlowski &
Ilgen, 2006). Despite the fact that training on complex tasks facilitates transfer, most of the training
literature tends to focus on relatively simple, routine tasks rather than complex, dynamic tasks
requiring adaptation (Kozlowski et al., 2001). Airlines operate in a dynamic environment requiring
adaptation. As a result, teams need to develop not
only routine expertise but also adaptive expertise
(Kozlowski, 1998). Adaptive expertise can be
facilitated by active learning experiences that are
naturalistic and adaptive. To this end, the simulations included unanticipated events that required
coordinated action.
Although some skills needed for effective
team performance are generic in nature and can
be trained in a variety of contexts, more specialized skills are task contingent and are best
trained in situations that closely match the task
environment (Cannon-Bowers, Tannenbaum,
Salas, & Volpe, 1995). Naturalistic adaptive
training of teams is best accomplished when
teams work in natural contexts and are faced
with situations that require them to monitor conditions and plan and collectively execute corrective actions (Kozlowski, 1998).
Team learning and performance can be
improved through the effective use of afteraction reviews (Villado & Arthur, 2013). A recent
1277
meta-analysis indicated that after-action reviews
can result in a 25% increase in team performance
(Tannenbaum & Cerasoli, 2013). Together, these
five techniques (onboarding, declarative knowledge training, procedural knowledge training,
simulations, and after-action reviews) incorporate sound training principles.
Team Training, Teamwork, and Team
Performance
Previous research suggests that the use of
effective training techniques can positively affect
team performance, teamwork, and emergent cognitive states that support teamwork. A metaanalysis indicates that team training is generally effective (Salas et al., 2008); team training
showed moderate relations with a wide range of
criteria, including team performance (ρ = .42),
process (ρ = .44), affective (ρ = .35), and cognitive (ρ = .42) outcomes. Team processes and cognitions are especially important for complex tasks,
interdependent tasks, and tasks where adaptation is needed (Hülsheger, Anderson, & Salgado,
2009; LePine, Piccolo, Jackson, Mathieu, & Saul,
2008). These findings underscore the importance
of teamwork and emergent cognitive states that
promote an understanding of task and teamwork
demands. It follows that training programs that
promote teamwork and relevant cognitive states
can promote effective team performance.
Because we utilized a multifaceted team
training program reflecting practices that have
been found to be effective (Gregory et al., 2013),
we hypothesize that the team training program
would be positively evaluated by participants
and would lead to positive changes in emergent
cognitive states, teamwork-related behavior, and
team performance. A recommended first step in
training evaluation is the assessment of participant evaluations (Kirkpatrick, 1979). Because
the training program was highly job related and
based on sound training theory, we expected
participants to believe that the training was valuable. Therefore we proposed the following:
Hypothesis 1: The training program would lead
to positive training evaluations.
A number of emergent cognitive states have
been identified that can serve as coordinating
1278
mechanisms to support effective teamwork
(Salas et al., 2005). These cognitive states
include collective efficacy and transactive memory. Transactive memory represents a shared
understanding of the knowledge distribution
within a group. Meta-analysis (DeChurch &
Mesmer-Magnus, 2010) indicated that transactive memory was related to team performance
(ρ = .44). A series of studies by Moreland
and colleagues (e.g., Moreland & Myaskovsky,
2000) indicated that training intact teams on an
assembly task led to the development of transactive memory systems and more effective team
performance. Hence, we posit the following:
Hypothesis 2: The team training program
would result in greater transactive memory.
Collective efficacy is a group’s shared belief
that it can collectively accomplish its task. A
meta-analysis by Stajkovic, Lee, and Nyberg
(2009) indicated a positive weighted average correlation between collective efficacy and team performance (G[r+] = .35) and especially for highly
interdependent tasks (G[r+] = .45). Collective
efficacy increases with task success and decreases
with failure experiences (Prussia & Kinicki,
1996). The training was expected to increase task
skill, but simulations became progressively more
difficult, making it difficult to confidently predict
the nature of changes in collective efficacy. Thus,
we examined changes in collective efficacy but
did not propose a directional hypothesis.
Research Question 1: Did the team training program affect the level of collective efficacy?
As previously indicated, meta-analytic evidence indicates that interactive team training
can promote teamwork (ρ = .44, Salas et al.,
2008). Team training has been shown to facilitate effective teamwork in multiple contexts,
such as simulated helicopter missions (Stout,
Salas, & Fowlkes, 1997), surgical simulations
(Paige et al., 2014), postoperative nursing teams
(Paull et al., 2013), and multiteam military
simulations (Firth, Hollenbeck, Miles, Ilgen, &
Barnes, 2015). Together these findings provide
strong evidence that team training can lead to
more effective teamwork, leading us to propose
the following:
December 2016 – Human Factors
Hypothesis 3: The team training would result in
more effective teamwork.
A meta-analysis (LePine et al., 2008) documented the consistent relationship between
teamwork and team performance. Both overall
teamwork (ρ = .31) and several facets of teamwork (e.g., backup behavior, goal specification,
conflict management) were related to team
performance. Meta-analytic evidence indicates
that team training can lead to team performance
improvements (Salas et al., 2008). Therefore,
we propose the following:
Hypothesis 4: The team training would result
in more effective group performance outcomes.
Method
Participants
Participants consisted of 394 students assigned
to 39 teams; all were aerospace (aviation science) students enrolled in an undergraduate,
senior-level capstone course. Although participation in training was a course requirement,
utilization of participant’s data for research
purposes was voluntary. This research complied
with the American Psychological Association
Code of Ethics and was approved by the Institutional Review Board at Middle Tennessee State
University. Informed consent was obtained from
each participant. The following concentrations
or specializations were represented: aerospace
administration, flight dispatch, maintenance
management, professional pilot, and aerospace
technology (i.e., engineering, computer science as they relate to aviation). Although these
senior-level students had completed significant
course work in their specializations, they possessed limited knowledge of the other specializations and had limited experience working
with students from the other specializations.
They were assigned to teams of approximately 10 people. There were five to six teams
in each semester. Each participant was assigned
to a specific position in multiteam simulations of
airline operations. The positions (described
later) were those typically found in a centralized
airline flight operations center as well as pilot
and an airport ramp tower coordinator positions.
Team Training
To the extent possible, participants were assigned
to positions that were consistent with their aviation specialization. Eighty-six percent were
assigned to a role matching their specialization
or utilizing skills closely related to their specialization. As described next, in all cases, participants received task-specific training on their
role within the simulation.
Training
Onboarding. Each semester, a 45-min orientation/onboarding session was conducted in a
classroom setting. It included a description of
the virtual airline, the series of simulations, the
team task, individual roles, and the need for
coordination. Participants were informed that
the simulations would take place in the Flight
Operations Unified Simulation (FOCUS) Lab
(described in following sections).
Task-specific training: Declarative knowledge. Each participant individually completed
two online modules: One module provided an
overview of the overall composition and function of the simulated flight operations center,
and the other module was specific to the participant’s position in the lab. In order to maximize
learning, each module had an online quiz that
had to be passed prior to the participant coming
to the hands-on training (Carpenter, 2012).
Task-specific training: Procedural knowledge. Each team participated in a session lasting
45 min to 1 hr; each team was taken into the lab,
and each member was provided with individual
instruction, demonstration, and supervised practice at his or her workstation. The purpose of this
session was to ensure that each participant
developed an understanding of his or her role
responsibilities and the technical knowledge to
do the job.
Simulations. Because airline operations require
coordinated actions among members of diverse
occupational specializations, we utilized a
series of team simulations. Each simulation
required a team of approximately 10 persons to
work collaboratively to conduct airline operations. These tasks included normal operations
needed for flight departures and arrivals (e.g.,
ensuring that planes were properly fueled,
attending to routine maintenance issues, and
ensuring that crews were available). Simulations
1279
also included nonroutine events that required
team adaptation. These events included unexpected maintenance issues, such as loss of cabin
pressure or an inoperable cargo door. They also
included in-flight passenger issues, such as a
medical emergency or an unruly passenger.
Additional situations could include a security
breach at an airport, inoperable refueling equipment, a sick crew member, or serious weather
issues. Effective responses to nonroutine events
could include not only actions to correct the
immediate problem but also actions to restore
normal conditions.
For example, if a plane experiences a serious
safety issue while in flight, the first concern is to
safely land, which may involve diversion to
another airport. If this event occurs, many downstream consequences may exist, and solutions
may be complex and multifaceted. Passengers
on the diverted flight must be rerouted, the
maintenance issue must be resolved, and a
replacement aircraft and flight crews will be
needed for later flight legs. Each of these problems may be difficult to resolve and may require
collaborative adaptive efforts. Because adaptive
team training may be best accomplished with a
series of active learning and reflection experiences, we utilized a series of simulations and
after-action reviews.
Simulation 1
During this 2.5-hr simulation, the participants collectively worked to operate the simulated airline. The team has operational control
of the airline and the ability to make operational
decisions. The airline is a regional carrier with
a fleet of 30 aircraft (50-passenger Canadian
Regional Jet [CRJ] aircraft commonly used
in regional airlines) serving 16 airports, two
of which serve as regional hubs with reserve
crews, reserve aircraft, and maintenance facilities. During the simulation, approximately 80
flight events (takeoffs and landings) occurred.
Much of the activity involved routine handling
of flights and required communication and
teamwork. In addition, nonroutine events (such
as severe weather, maintenance issues, or other
problems requiring attention) occurred and further increased the need for information transfer,
coordinated action, and adaptation.
1280
Figure 1. FOCUS Lab components.
After-Action Review 1. Following the first simulation, facilitated after-action reviews were conducted. In preparation for the after-action review,
participants individually completed a form about
successful and unsuccessful events and their perceptions of team behaviors that contributed to
these outcomes (Tannenbaum, Beard, & Cerasoli,
2013). During the after-action review, the group
participated in a facilitated discussion of positive
and negative events and opportunities for improvement. This session typically lasted approximately
1 hr.
Simulations 2 and 3
These simulations were similar to the first
simulation. They involved the same flight schedule, but a different set of nonroutine events were
presented each time. Consistent with the training goals of the studies, the number and/or difficulty of nonroutine events tended to increase
across the three simulations. The second and
third after-action reviews followed the same
format as the first, but in the third, there was a
focus on lessons learned that participants could
take with them into their careers and advice they
would give students starting the lab.
Setting and Positions
The simulations were conducted in the
FOCUS Lab, a four-room facility (see Figure 1).
This facility includes a flight operations center,
a ramp control tower, a pseudopilot room, and
a CRJ flight simulator. This setting mirrors the
task environments of a regional airline.
December 2016 – Human Factors
Flight operations center. This room housed
multiple workstations. The flight operations
coordinator had the most central role and had
final decision-making authority for most matters
related to the operation of flights. In order to
make effective decisions, this person needed to
utilize information from all other participants in
the simulation, including those in other rooms.
The flight operations coordinator could communicate with all individuals in the simulation and
was the primary boundary spanner between the
flight operations center team and other positions
(pseudopilot, CRJ pilot and copilot, and ramp
tower coordinator).
The other positions in this room (a) ensured
that flights were properly loaded, they had sufficient fuel, and passengers and cargo were effectively managed (flight planning); (b) tracked the
flights and maintained and updated the flight
schedule data (flight data and tracking); (c) considered weather conditions (weather monitoring);
(d) ensured that flight crews did not exceed legal
duty time limitations (crew scheduling); and (e)
ensured that aircraft were properly maintained and
safe for flight (maintenance planning and maintenance scheduling). Personnel in the flight operations center were seated around the rim of a double
row of long tables. This arrangement allowed for
face-to-face communication between these positions, but headset and text communication was
also available.
Six large-screen monitors were wall mounted;
three were behind each long side of the tables so
that each side displayed the flight schedule, a
radar view of the flights in progress, and a
weather map. These displays provided real-time
information during the simulation. In addition,
each workstation was equipped with a computer
and two monitors to allow access to multiple
sources of data relevant to that position.
Ramp control tower. The ramp control tower
was located in an adjacent room and simulated the
operation of one of the airline’s hub airports
(Nashville). The ramp tower coordinator directed
arriving and departing flights to appropriate taxiways and gates and provided notification to the
flight operations coordinator when a plane was
ready for release from the gate area. This room
contained three wall-mounted large-screen monitors providing computer-generated, panoramic,
Team Training
real-time views of the gates, runways, and taxiways. Another display showed a radar view of
flights preparing for landing and takeoff, and a
computer allowed the operator to direct the planes
to gates and taxiways. The ramp tower coordinator could communicate with the flight operations
coordinator via headset and text.
Pseudopilot room. The pseudopilot room
consisted of a single workstation where flights
from locations other than Nashville were started
by the pseudopilot. (As described earlier, flights
from the Nashville airport were requested from
the ramp tower.) This position could also divert
aircraft if so directed and could report any issues
that arose with these flights. This workstation
consisted of a single computer terminal. Voice
and text communication was available with
positions in the flight operations center and with
the ramp tower coordinator.
CRJ simulator. The CRJ simulator was a
fully functional simulation of an aircraft (CRJ200) cockpit area. It was located offsite and was
operated by the CRJ pilot and CRJ copilot. The
CRJ simulator represented one of the planes in
the airline’s fleet, and (unlike the pseudopilot)
the CRJ crew had full operational control of the
aircraft. Voice and text communication was
available with the flight operations center,
weather, and the ramp control tower. Additional
details about the FOCUS Lab and the individual
positions and additional methodological details
are available in the supplementary materials at
http://hfs.sagepub.com/supplemental.
Data Collection Procedures
Data were collected during seven semesters. Procedures differed somewhat between the
semesters. During the first two semesters (10
teams), each team participated in two simulations, and the CRJ simulator was not utilized.
During the other semesters (29 teams), the CRJ
simulator was utilized, and a third simulation
and after-action review were added. Participants
completed research instruments at individual
computer stations in a computer lab reserved for
their use. All sessions involved the operation of
the same airline routes with the same resources,
but weather conditions varied and events, such
as maintenance problems or other issues, were
manipulated. The simulations were designed
1281
to create the feel of a work shift much like participants would experience upon entering the
workforce in commercial aviation.
Measures
Affective reactions. A five-item scale was
used to assess participants’ evaluation of the
training experiences. Following the final simulation, participants indicated the extent of
agreement with statements that the experiences were valuable (e.g., helped understand
the work of other specializations, helped
understand the need for teamwork, helped prepare them for job demands). The response
scale was anchored by strongly disagree (1)
and strongly agree (6).
Emergent cognitive states. Two emergent
cognitive states were examined. These included
transactive memory and collective efficacy.
Transactive memory: Transactive memory was
assessed using a 15-item scale developed by
Lewis (2003) in which team members (participants) evaluated the awareness of the
knowledge and responsibilities of the various professional specializations found in
their FOCUS Lab team, how credible they
believed that knowledge was, and how well
the team coordinated their work. A 5-point
response scale was used that ranged from
strongly disagree (1) to strongly agree (5).
This measure was collected both following
the task-specific training and after the last
simulation.
Collective efficacy: This 10-item scale was
based on the collective efficacy scale by
Quiñones (1995). The 5-point response
scale ranged from strongly disagree (1) to
strongly agree (5). The original scale was
modified for our research to measure participants’ beliefs about their team’s future performance during an upcoming FOCUS Lab
simulation. This measure was first collected
following the task specific training and also
after each simulation.
Behaviors. Two types of measures were used
to assess participant behavior. These were participant self-ratings of teamwork and observer
ratings of teamwork.
1282
Self-rated teamwork: Teamwork was assessed
using a 30-item self-report teamwork scale
developed by Mathieu and Marks and
based on Marks et al. (2001). Each item
was phrased as the extent to which the team
actively worked to perform various teamwork behaviors; the response scale ranged
from 1 (not at all) to 5 (to a very great
extent). The scale yielded an overall teamwork score, scores for teamwork during
action and transition (planning and reflection) phases, and scores for interpersonal
behavior. Participants completed this measure after each simulation.
Observer-rated teamwork: Observers (members
of the research team) rated teams on teamwork behaviors using a 10-item, behaviorally anchored scale developed specifically
for this research. The measure included an
overall score and three subscales: problem
solving (four items), coordination (two items),
and information utilization (four items). The
observers used a 7-point rating scale where a
rating of 1 reflected behaviors expected at the
trainee level, a rating of 3 reflected behaviors
expected at the developing level, a rating of
5 reflected behaviors expected at the experienced level, and a rating of 7 reflected behaviors expected at the professional level. Ratings
were collected after each simulation.
Team performance. For each FOCUS Lab
simulation, the system software provided two
objective measures of team performance.
Delay time: Delay time was calculated for
each flight during a simulation by subtracting the time difference between the scheduled arrival and actual arrival times of each
flight. Delay time reflects the total hours of
delay pooled across all flights scheduled
during a simulation.
Financial loss: Financial loss was measured in
dollars. Each simulation began with a possible revenue value based on the money generated by the number of passengers and the
number of flights (approximately $400,000
per simulation). Any time delay in a flight
resulted in a financial penalty based on the
amount of time delay and the number of
December 2016 – Human Factors
passengers on a flight using an airline industry–
based formula. These measures (time delay
and financial loss) accrued over the course of a
simulation. These data were collected for each
simulation and presented to the teams during
the next after-action review.
Results
Unless otherwise noted, data were examined
using repeated-measures ANOVAs contrasting initial and final measures. See Table 1 for
means and standard deviations of initial and
final measures.
Affective Reactions
The five-item affective reactions scale was
internally consistent (α = .90). Consistent with
Hypothesis 1, a single-sample t test revealed
that the mean rating (M = 5.21) reflected positive evaluations, t(312) = 34.48, p < .001. Emergent Cognitive States Transactive memory. Hypothesis 2 stated that training would lead to higher levels of transactive memory. The transactive memory scale showed adequate internal consistency (α = .73). The intraclass correlation (ICC1) represents an assessment of the variance attributable to group membership (Bliese, 2000, p. 354). Prior to simulations, there was essentially no group-level variance (ICC1 = .003); following simulations, some degree of group-level variance was apparent (ICC1 = .06). Participants were senior aerospace majors and, therefore, would be expected to have partially developed transactive memory systems. In fact, participants did initially display a general awareness of the knowledge and responsibilities of various professional specializations (M = 3.70). Consistent with Hypothesis 1, higher levels of transactive memory were observed following training (M = 4.04), F(1, 33) = 83.32, p < .001, η2 = .72). Together these results indicate that although there was not a great deal of withingroup agreement about the extent of transactive memory, the initially moderate level of transactive memory was somewhat enhanced following the simulations and related experiences. Collective efficacy. Research Question 1 concerned the effects of team training on collective Team Training 1283 Table 1: Initial and Final Descriptive Statistics Initial Measure Affective reactions Transactive memory Collective efficacy Self-rated teamwork (overall) Action processes Transition processes Interpersonal processes Observer-rated teamwork (overall) Problem solving Coordination Information utilization Delay time (hours) Financial loss (thousands) M 3.70 4.16 3.73 3.63 3.57 4.02 3.32 3.22 3.28 3.46 10.80 $35.9 efficacy. Internal consistency was high (α = .88). Prior to the simulations, there was little withingroup agreement (ICC1 = .02), but agreement increased following the simulations (ICC1 = .13). In order to evaluate Research Question 1, a repeated-measures (pretest-posttest) ANOVA was conducted to compare the level of collective efficacy following initial positional training and following simulations. It revealed that collective efficacy did not show consistent change as a result of the simulations, F < 1. In order to determine if teams differed in level of collective efficacy, a pair of one-way ANOVAs (one for initial and one for final collective efficacy) contrasted the level of collective efficacy across teams. Initially, collective efficacy did not differ across teams, F < 1, but following the simulations and related experiences, efficacy did differ across teams, F(33, 255) = 2.31, p < .001, η2 = .23. Together these findings suggest that although there was no consistent overall change in collective efficacy, it appears that the training may have positively affected the efficacy of some teams while negatively affecting the efficacy of other teams. Teamwork Hypothesis 3 predicted that team training would result in more effective teamwork behavior. Final SD 0.12 0.19 0.28 0.29 0.35 0.27 0.76 0.81 0.81 0.74 4.49 $14 M 5.21 4.04 4.15 4.06 3.96 3.98 4.25 4.57 4.44 4.55 4.70 6.41 $24.1 SD 0.88 0.20 0.35 0.32 0.33 0.37 0.31 1.01 1.07 1.06 0.93 4.14 $16 Teamwork was assessed both by self-ratings and by observer ratings. Self-rated teamwork. The self-rated teamwork scale showed high internal consistency (α = .96) and a moderate degree of within-group agreement (initial ICC1 = .09; postsimulation ICC1 = .15). A pretest-posttest ANOVA of the overall teamwork scale indicated that teamwork showed a modest increase from the initial simulation (M = 3.73) to the final simulation (M = 4.06), F(1, 37) = 53.10, p < .001, η2 = .59. Each of the three facets of teamwork, action processes, F(1, 37) = 46.01, p < .001, η2 = .55; transition processes, F(1, 37) = 49.61, p < .001, η2 = .57; and interpersonal processes, F(1, 37) = 34.09, p < .001, η2 = .48 increased as well. Observer-rated teamwork. Examination of observer ratings of teamwork (α = .96) revealed large improvements in teamwork. Observer ratings of teamwork increased from 3.32 to 4.57 for the overall teamwork scale, F(1, 26) = 34.67, p < .001, η2 = .57. Likewise, observer ratings of each component of teamwork increased from the first to the final simulation: problem solving, F(1, 26) = 28.32, p < .001, η2 =.52; coordination, F(1, 26) = 31.69, p < .001, η2 = .55; and information utilization, F(1, 26) = 41.91, p < .001, η2 = .62. Consistent with Hypothesis 3, both participantrated and observer-rated teamwork increased with team training. 1284 December 2016 - Human Factors Outcome Measures Delay time. Mean delay time decreased by more than 4 hr from a mean of 10.80 hr on the initial simulation to 6.41 hr on the final simulation, F(1, 32) = 17.68, p = .001, η2 = .36. These results indicate that team performance improved as training progressed despite the fact that later simulations were more challenging. Financial loss. Financial losses due to flight delays decreased by more than 32% ($11,721). The mean loss on the initial simulation was $35,865, whereas the mean loss on the final simulation was $24,145, F(1, 32) = 10.13, p < .01, η2 = .24. Decreases in both delay time and financial losses provide support for Hypothesis 4 that training would lead to more effective team performance. Discussion Results provided support for all four hypotheses. Consistent with Hypothesis 1, participants perceived the training to be valuable. Likewise, training affected transactive memory, teamwork, and performance. Training positively affected transactive memory. Because participants were senior aerospace majors, it is reasonable to expect that they would have a general understanding of the knowledge and responsibilities of various specializations. This appeared to be the case; initial levels of transactive memory were moderately high and essentially the same as those observed in MBA teams completing yearlong consulting projects (Lewis, 2003). Nevertheless, we expected the interactive team training to increase transactive memory. Supporting Hypothesis 2, the training led to higher transactive memory scores. This finding is consistent with previous findings that intact team training (Lewis, 2003; Moreland & Myaskovsky, 2000) and task experience in intact teams (Littlepage, Robison, & Reddington, 1997) can lead to the development of transactive memory. Thus, the team training experiences allowed team members to become more aware of the knowledge, skills, abilities, and responsibilities of other team members. Both member ratings and observer ratings indicated that teamwork improved across simulations, providing support for Hypothesis 3. Improvements in teamwork are important because meta-analytic findings confirm that various teamwork behaviors are related to effective team performance (LePine et al., 2008). This meta-analysis also indicated that although generally related to performance, teamwork was somewhat more important with large groups and interdependent tasks. These factors were present in the current simulations (and the airline flight operations centers they were designed to reflect) as we used relatively large groups of specialists working on highly interdependent tasks in a time-sensitive, multiteam environment. The reduction in delay times across simulations and increased revenues indicate that team performance improved as a result of the training and provides support for Hypothesis 4. Although it is likely that some of the performance improvements were the result of increased individual task knowledge and skill acquired during the simulations, the enhanced transactive memory and teamwork suggest an additional factor: Participants learned to work more effectively as a team. Team cognition and teamwork are important mediators of group performance (Ilgen, Hollenbeck, Johnson, & Jundt, 2005; LePine et al., 2008). Our findings suggest that highfidelity team simulations led to a more refined understanding of the nature of team member knowledge, more effective teamwork, and higher levels of team performance. Research Question 1 examined potential changes in collective efficacy. Because team performance improved across simulations, it might be anticipated that participation in the simulations would also increase feelings of efficacy; this was not the case. Comparison of presimulation and postsimulation levels of collective efficacy revealed no change. Shea and Howell (2000) found that individuals had unrealistic initial performance expectations. It is possible that group members’ initial levels of collective efficacy were unrealistically high and that performance feedback resulted in more realistic levels. Collective efficacy can increase or decrease as a result of positive or negative feedback about group performance (Prussia & Kinicki, 1996; Tindale, Kulik, & Scott, 1991). Thus, it is possible that the lack of change in collective efficacy could be the result of offsetting Team Training processes of revision of initially unrealistic initial performance expectations and intrinsic and explicit feedback about performance gains deriving from increased task experience. Limitations This project had both training and research objectives, which can require trade-offs. Training objectives are best served with progressively more difficult scenarios, whereas standardization across scenarios is more compatible with research objectives. In the present study, consistent with training objectives, scenarios tended to become more challenging across simulations. Although this design is not consistent with the ideal of standardization desired for research, findings of improved teamwork and performance are more impressive, considering the increasing complexity of simulations. A related issue stems from the extreme high fidelity of the simulations, which is both a strength and a limitation of the study. The positive consequences of the high fidelity are greater mundane and psychological realism and potentially greater transfer of training (Cannon-Bowers et al., 1995). A negative consequence of the high level of fidelity is the lack of standardization. In order to reduce treatment contamination, a wide variety of nonroutine events was used, and they frequently varied across simulations. Attempts were made to roughly equate the difficulty of nonroutine events during each round of simulations, but we do not have evidence of equivalence. The fidelity of the simulations makes it extremely hard to equate the difficulty level of nonroutine events, because the difficulty of an event is often largely determined by the adequacy of the team’s initial response. That is, even with a simple problem, issues can quickly proliferate if the team does not respond properly. The study utilized a pretest-posttest design and is subject to internal validity threats, such as maturation and history (Shadish, Cook, & Campbell, 2002). For example, it is possible that our results do not derive from the training program but instead may reflect effects of group development resulting from the interaction with the team members or a general familiarity with the airline simulation. Although these alterna- 1285 tive explanations could be evaluated with the addition of control groups, a variety of factors made the creation of appropriate control conditions difficult. Most measures were based on behavior, cognitive states, or performance in, or deriving directly from, the simulations. It would be problematic to assess these variables using individuals without the simulation experience. Even if it were possible to develop meaningful control conditions, that would require control participants with the same level of knowledge and skills as the participants in the training condition. We had a limited pool of senior aerospace majors, and assigning some to control conditions would greatly reduce the number of participants assigned to the training program and would also be inconsistent with the ethical principle of justice by withholding an effective training intervention (Hoyle, Harris, & Judd, 2002). Thus, although a stronger experimental design would have been ideal, it was not practical in this study. Future Directions Although results show the effects of the combined training program, they do not provide evidence of the relative importance of the various components. Previous research suggests that each of the components can be effective (Banks & Millward, 2007; Salas et al., 2010; Snell, 2006; Tannenbaum & Cerasoli, 2013), but it is unclear whether each component adds incremental benefit to the various outcomes. It may be that one component is of primary importance, but it is also possible that all have meaningful impact or that the various components have differential impact on various outcomes. That is, one facet may have its greatest effect on emergent cognitive states whereas another component may have a greater impact on teamwork behavior or performance. Although the study design does not allow for the assessment of the contribution of individual training components, the examination of the relative importance of various facets of the training program is worthy of attention. Transfer of training to the work context is worthy of examination. Although it is clear that senior aviation students developed enhanced transactive memory and that teamwork and team 1286 December 2016 - Human Factors performance improved, these effects were noted only within the training context. Work to examine the postgraduation transition to aviation careers would be a useful extension. Acknowledgments Conclusions and Applications Findings provide evidence that a multifaceted training program that features high-fidelity team simulations can lead to enhanced teamwork, transactive memory, and enhanced team performance. The current research extends work on team training by showing the positive effects of team training for relatively large crossfunctional teams conducting extremely complex tasks requiring adaptation and operating within a multiteam context. The approach used here can serve as a model for initial and recurring training at commercial airlines and in military aviation. Some facets of aviation rely extensively on training techniques like those utilized here. Although pilots in training and professional pilots undergo extensive simulator training and after-action reviews, the use of interactive training across aviation disciplines is rare. The extensive use of a comprehensive set of validated training techniques in crossfunctional aviation teams or in multiteam systems may prove to be an effective way to facilitate coordinated performance within an airline. Such training can facilitate onboarding of new hires or refresher training or can serve as a component of selection programs for individuals seeking entry into a variety of aviation professions. The approaches used here also have the potential to be used to improve teamwork and performance of teams in other settings that require coordination. Although training programs such as this one may be of use in many team settings, they may be especially useful in cross-functional teams and in multiteam systems. The utility of team-based simulations has been demonstrated in various fields, such as health care, nuclear power, and the military (Bogenstätter et al., 2009; Stachowski, Kaplan, & Waller, 2009; Tschan et al., 2009). However, the greater use of comprehensive programs consisting of multiple training components of known effectiveness may be warranted. The development of the simulation lab was supported by NASA Grant No. NNX10AI11G. We thank Paul Carlson, Durant Bridges, Evan Lester, and Joe Cooper for technical assistance and Morgan Pearn, Amber Fritsch, Emily Sanders, and Jennifer Henslee for assistance with data collection and management. Key Points •• A comprehensive training program featuring simulations requiring cross-functional coordination can enhance team performance, teamwork behavior, and transactive memory. •• Entry-level aviation professionals can benefit from task-relevant training involving collaboration among aviation specializations. Supplementary Materials The online supplementary material is available at http://hfs.sagepub.com/supplemental. References Ball, M., Barnhart, C., Dresner, M., Hansen, M., Neels, K., Odoni, A., … Zou, B. (2010). Total delay impact study: A comprehensive assessment of the costs and impacts of flight delay in the United States. National Center of Excellence for Aviation Operations Research. Retrieved from http://www.nextor.org/ pubs/TDI_Report_Final_11_03_10.pdf Banks, A. P., & Millward, L. J. (2007). Differentiating knowledge in teams: The effect of shared declarative and procedural knowledge on team performance. Group Dynamics: Theory, Research, and Practice, 11, 95–106. doi:10.1037/10892699.11.2.95 Beer, M. (2013). Silent killers of team performance: How honest, collective, and public conversations can overcome them. In E. Salas, S. I. Tannenbaum, D. J. Cole, & G. Latham (Eds.), Developing and enhancing teamwork in organizations: Evidence-based best practices and guidelines (pp. 27–47). San Francisco, CA: Jossey-Bass. Bienefeld, N., & Grote, G. (2014). Shared leadership in multiteam systems: How cockpit and cabin crews lead each other to safety. Human Factors, 56, 270–286. doi:10:1177/0018720813488137 Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass. Bogenstätter, Y., Tschan, F., Semmer, N. K., Spychiger, M., Breuer, M., & Marsch, S. (2009). How accurate is information transmitted to medical professionals joining a medical emergency? A simulator study. Human Factors, 51, 115–125. doi:10.1177/0018720809336734 Cannon-Bowers, J. A., Rhodenizer, L., Salas, E., & Bowers, C. A. (1998). A framework for understanding pre-practice conditions Team Training and their impact on learning. Personnel Psychology, 51, 291–320. doi:10.1111/j.1744-6570.1998.tb00727.x Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. F. (1995). Defining competencies and establishing team training requirements. In R. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (pp. 333–380). San Francisco, CA: Jossey Bass. Carpenter S. K. (2012). Testing enhances the transfer of learning. Current Directions in Psychological Science, 21, 279–283. doi:10.1177/0963721412452728 Caruso, H. M., & Woolley, A. W. (2008). Harnessing the power of emergent interdependence to promote diverse team collaboration. In K. W. Phillips (Ed.), Diversity and groups (pp. 245–266). Stamford, CT: JAI Press. DeChurch, L. A., & Marks, M. A. (2006). Leadership in multiteam systems. Journal of Applied Psychology, 91, 311–329. doi:10.1037/0021-9010.91.2.311 DeChurch, L. A., & Mesmer-Magnus, J. R. (2010). The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology, 95, 32–53. doi:10.1037/a0017328 Devine, D. J., Clayton, L. D., Philips, J. L., Dunford, B. B., & Melner, S. B. (1999). Teams in organizations: Prevalence, characteristics, and effectiveness. Small Group Research, 30, 678–711. doi:10.1177/104649649903000602 Federal Aviation Administration. (2004). Crew resource management training (Advisory Circular 120-51E). Retrieved from http://www.faa.gov/documentLibrary/media/Advisory_Circular/ AC120-51e.pdf Firth, B. M., Hollenbeck, J. R., Miles, J. E., Ilgen, D. R., & Barnes, C. M. (2015). Same page, different books: Extending representational gaps theory to enhance performance in multiteam systems. Academy of Management Journal, 58, 813–835. doi:10.5465/amj.2013.0216 Gregory, M. E., Feitosa, J., Driskell, T., Salas, E., & Vessey, W. B. (2013). Designing, delivering, and evaluating team training in organizations. In E. Salas, S. I. Tannenbaum, D. J. Cohen, & G. Latham (Eds.), Developing and enhancing teamwork in organizations: Evidence-based best practices and guidelines (pp. 441–487). San Francisco, CA: Jossey-Bass. Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The evolution of crew resource management training in commercial aviation. International Journal of Aviation Psychology, 9, 19–32. doi:10.1207/s15327108ijap0901_2 Howard, C. E. (2011). Simulation and training: Expecting the unexpected. Military and Aerospace Electronics, 22(11), 12–23. Hoyle, R. H., Harris, M. J., & Judd, C. M. (2002). Research methods in social relations (7th ed.). Belmont, CA: Wadsworth. Hülsheger, U. R., Anderson, N., & Salgado, J. F. (2009). Teamlevel predictors of innovation at work: A comprehensive metaanalysis spanning three decades of research. Journal of Applied Psychology, 94, 1128–1145. doi:10.1037/a0015978 Ilgen, D. R., Hollenbeck, J. R., Johnson, M., & Jundt, D. (2005). Teams in organizations: From input-process-output models to IMOI models. Annual Review of Psychology, 56, 517–543. doi:10.1146/annurev.psych.56.091103.070250 Kanki, B. G., Helmreich, R. L., & Anca, J. (2010). Crew resource management. San Diego, CA: Academic Press. Kirkpatrick, D. L. (1979). Techniques for evaluating training programs. Training & Development Journal, 33, 78–92. Kozlowski, S. W. J. (1998). Training and developing adaptive teams: Theory, principles, and research. In J. A. CannonBowers & E. Salas (Eds.), Making decisions under stress: 1287 Implications for individual and team training (pp. 115–153). Washington, DC: American Psychological Association. Kozlowski, S. W. J., & Ilgen, D. R. (2006). Enhancing the effectiveness of work groups and teams. Psychological Science in the Public Interest, 7, 77–124. doi:10.1111/j.15291006.2006.00030.x Kozlowski, S. W. J., Toney, R. J., Mullins, M. E., Weisband, D. A., Brown, K. G., & Bell, B. S. (2001). Developing adaptability: A theory for the design of integrated-imbedded training systems. In E. Salas (Ed.), Advances in human performance and cognitive engineering research (Vol. 1, pp. 59–123). Amsterdam, Netherlands: JAI/Elsevier Science. Lawler, E. E., III, Mohrman, S. A., & Benson, G. (2001). Organizing for high performance. San Francisco, CA: Jossey-Bass. LePine, J. A., Piccolo, R. F., Jackson, C. L., Mathieu, J. E., & Saul, J. R. (2008). A meta-analysis of teamwork processes: Tests of a multidimensional model and relationships with team effectiveness criteria. Personnel Psychology, 61, 273–307. doi:10.1111/ j.1744-6570.2008.00114.x Lewis, K. (2003). Measuring transactive memory systems in the field: Scale development and validation. Journal of Applied Psychology, 88, 587–604. doi:10.1037/0021-9010.88.4.587 Littlepage, G., Robison, W., & Reddington, K. (1997). Effects of task experience and group experience on group performance, member ability, and recognition of expertise. Organizational Behavior And Human Decision Processes, 69, 133–147. doi:10.1006/obhd.1997.2677 Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26, 356–376. doi:10.2307/259182607 Marks, M. A., Zaccaro, S. J., & Mathieu, J. E. (2000). Performance implications of leader briefings and team-interaction training for team adaptation to novel environments. Journal of Applied Psychology, 85, 971–986. doi:10.1037/0021-9010.85.6.971 Mathieu, J. E., Marks, M. A., & Zaccaro, S. J. (2001). Multiteam systems. In N. Anderson, D. Ones, H. K. Sinagil, & C. Visesvaran (Eds.), International handbook of work and organizational psychology (pp. 289–313). London, UK: Sage. Merket, D. C., Bergondy, M. L., & Salas, E. (2000). Making sense out of team performance errors in military aviation environments. Transportation Human Factors, 1, 231–242. Moreland, R. L., & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes, 82, 117–133. doi:10.1006/ obhd.2000.2891 O’Connor, P., Campbell, J., Newon, J., Melton, J., Salas, E., & Wilson, K. A. (2008). Crew resource management training effectiveness: A meta-analysis and some critical needs. International Journal of Aviation Psychology, 18, 353–368. doi:10.1080/10508410802347044 Paige, J. T., Garbee, D. D., Kozmenko, V., Yu, Q., Kozmenko, L., Yang, T., … Swartz, W. (2014). Journal of the American College of Surgeons, 218, 140–149. doi:10.1016/j.amcollsurg.2013.09.006 Paull, D. E., DeLeeuw, L. D., Wolk, S., Paige, J. T., Neily, J., & Mills, P. D. (2013). The effect of simulation-based crew resource management training on measurable teamwork and communication among interprofessional teams caring for postoperative patients. Journal of Continuing Education in Nursing, 44, 516–524. doi:10.3928/00220124-20130903-38 1288 Prussia, G. E., & Kinicki, A. J. (1996). A motivational investigation of group effectiveness using social-cognitive theory. Journal of Applied Psychology, 81, 187–198. doi:10.1037/0021-9010.81.2.187 Quiñones, M. A. (1995). Pretraining context effects: Training assignment as feedback. Journal of Applied Psychology, 80, 226–238. doi:10.1037/0021-9010.80.2.226 Salas, E., Bowers, C. A., & Rhodenizer, L. (1998). It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. International Journal of Aviation Psychology, 8, 197–208. doi:10.1207/s15327108ijap0803_2 Salas, E., Cooke, N. J., & Gorman, J.C. (2010). The science of team performance: Progress and the need for more. Human Factors, 52, 344–346. doi:10.1177/0018720810374614 Salas, E., DiazGranados, D., Klein, C., Burke, C. S., Stagl, K. C., Goodwin, G. F., & Halpin, S. M. (2008). Does team training improve team performance? A meta-analysis. Human Factors, 50, 903–933. doi:10.1518/00187208X375009 Salas, E., Sims, D. E., & Burke, C. S. (2005). Is there a “big five” in teamwork? Small Group Research, 36, 555–599. doi:10.1177/1046496405277134 Salas, E., Wilson, K. A., Burke, C. S., & Wightman, D. C. (2006). Does crew resource management training work? An update, an extension, and some critical needs. Human Factors, 48, 392– 412. doi:10.1518/001872006777724444 Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. Shappell, S., Detwiler, C., Holcomb, K., Hackworth, C., Boquet, A., & Wiegmann, D. A. (2007). Human error and commercial aviation accidents: An analysis using the Human Factors Analysis and Classification System. Human Factors, 49, 227–242. doi:10.1518/001872007X312469 Shea, C. M., & Howell, J. M. (2000). Efficacy–performance spirals: An empirical test. Journal of Management, 26, 791–812. doi:10.1016/S0149-2063(00)00056-8 Shuffler, M. L., Jimenez-Rodriguez, M., & Kramer, W. S. (2015). The science of multiteam systems: A review and future research agenda. Small Group Research, 46, 659–699. doi:10.11177/1046496415603455 Snell, A. (2006). Researching onboarding best practice: Using research to connect onboarding processes with employee satisfaction. Strategic HR Review, 5, 32–35. Stachowski, A. A., Kaplan, S. A., & Waller, M. J. (2009). The benefits of flexible team interaction during crises. Journal of Applied Psychology, 94, 1536–1543. doi:10.1037/a0016903 Stajkovic, A. D., Lee, D., & Nyberg, A. (2009). Collective efficacy, group potency, and group performance: Meta-analysis of their relationships and test of a mediation model. Journal of Applied Psychology, 94, 814–828. doi:10.1037/a0015659 Stout, R. J., Salas, E., & Fowlkes, J. E. (1997). Enhancing teamwork in complex environments through team training. Group Dynamics: Theory, Research, & Practice, 1, 169–182. Tannenbaum, S. I., Beard, R. L., & Cerasoli, C. P. (2013). Conducting team debriefings that work: Lessons from research and practice. In E. Salas, S. I. Tannenbaum, D. J. Cohen, & G. Latham (Eds.), Developing and enhancing teamwork in organizations: Evidence-based best practices and guidelines (pp. 488–519). San Francisco, CA: Jossey-Bass. Tannenbaum, S. I., & Cerasoli, C. P. (2013). Do team and individual debriefs enhance performance? A meta-analysis. Human Factors, 55, 231–245. doi:10.1177/0018720812448394 December 2016 - Human Factors Tindale, R., Kulik, C. T., & Scott, L. A. (1991). Individual and group feedback and performance: An attributional perspective. Basic and Applied Social Psychology, 12, 41–62. doi:10.1207/ s15324834basp1201_4 Tschan, F., Semmer, N. K., Gurtner, A., Bizzari, L., Spychiger, M., Breuer, M., & Marsch, S. U. (2009). Explicit reasoning, confirmation bias, and illusory transactive memory: A simulation study of group medical decision making. Small Group Research, 40, 271–300. doi:10.1177/1046496409332928 Villado, A. J., & Arthur, W., Jr. (2013). The comparative effect of subjective and objective after-action reviews on team performance on a complex task. Journal of Applied Psychology, 98, 514–528. doi:10.1037/a0031510 Zaccaro, S. J., Marks, M. A., & DeChurch, L. A. (2012). Multiteam systems: An introduction. In S. J. Zaccaro, M. A. Marks, & L. A. DeChurch (Eds.), Multiteam systems: An organizational form for dynamic and complex environments (pp. 3–32) New York, NY: Routledge. Glenn E. Littlepage is a professor in the Psychology Department at Middle Tennessee State University. He received his PhD in social psychology in 1974 from Kansas State University. Michael B. Hein is a professor in the Psychology Department at Middle Tennessee State University and the director of the Center for Human Resource and Organizational Effectiveness. He received his PhD in industrial/organizational psychology in 1990 from Georgia Institute of Technology. Richard G. Moffett III is an associate professor in the Psychology Department at Middle Tennessee State University and a senior consultant at the Center for Human Resource and Organizational Effectiveness. He received his PhD in industrial/ organizational psychology in 1996 from Auburn University. Paul A. Craig is a professor in the Aerospace Department at Middle Tennessee State University. He received his EdD in curriculum and instruction in 1998 from Tennessee State University. Andrea M. Georgiou is an associate professor in the Aerospace Department at Middle Tennessee State University. She received her PhD in educational psychology in 2014 from Capella University. Date received: September 19, 2015 Date accepted: July 20, 2016 Cogn Tech Work (2017) 19:263–277 DOI 10.1007/s10111-017-0421-7 ORIGINAL ARTICLE A virtual reality flight simulator for human factors engineering Matthias Oberhauser1 • Daniel Dreyer2 Received: 19 April 2016 / Accepted: 22 June 2017 / Published online: 15 July 2017 Springer-Verlag London Ltd. 2017 Abstract This research presents a virtual reality flight simulator (VRFS) that combines the advantages of desktop simulations and hardware mock-ups, i.e., the flexibility of a desktop flight simulation with the level of immersion close to a full flight simulator. In contrast to similar existing virtual reality flight simulators, the presented system focuses on human factors (HF) engineering and is used for evaluating flight decks in an early phase of the design process. Hence, HF tools that are based on HF methods have been integrated; applying these methods requires collecting objective (e.g., eye tracking, physiological data, head and finger movements) as well as subjective data (e.g., questionnaires). In this paper, three user studies are presented that demonstrate the application of the integrated HF methods and the general usability of the system. These studies have been conducted as part of human–machine interface (HMI) development projects and range from basic cognitive research to HMI evaluations using realistic scenarios. The user studies indicate that HF engineering with the help of this system is possible and a feasible alternative to other means of evaluation. Yet, the abilities are limited due to technological and physiological constraints. This is why the scope of the VRFS lies between desktop simulations and a full hardware mock-up and cannot replace either of those. However, the presented studies show that the system can provide reliable information on the interaction with HMI. Thus, it is a reliable low-cost addition in & Matthias Oberhauser matthias.oberhauser@student.tugraz.at 1 Technical University of Graz, 8010 Graz, Austria 2 Airbus Defence and Space GmbH, 85077 Manching, Germany the early development process of cockpit human machine interaction technologies when it comes to HF evaluations. Keywords Virtual reality Human factors engineering Virtual reality flight simulation Flight deck design 1 Introduction Modern commercial aircraft cockpits are safety-critical products with advanced and well-researched user interfaces. Due to strict quality standards for both hardware and software, the manufacturing costs and, consequently, the final product prices are high. As in every other complex product development cycle, adding changes to a mature product causes even higher costs and can lead to design compromises, which often results in expensive products with less-than-perfect user interfaces. In order to avoid such changes, knowledge about the final product’s properties should be generated as early as possible. Thus, the development of human machine interfaces in flight deck design requires feedback on human factors, i.e., ergonomics, usability, and cognitive aspects at a very early stage of the design process (Reuzeau and Nibbelke 2004). Depending on the stage of development of a product, the HF evaluation methods and prototypes should be chosen carefully in order not to tie up crucial resources (Kelly 2004). First, it is necessary to select the right level of simulation fidelity, corresponding to the maturity of the prototype and the development stage. The second step is to determine the adequate human factors method. Some methods use subjective questionnaires (e.g., NASA-TLX and SART) while others collect objective data such as physiological parameters (e.g., gaze behavior and heart rate), or performance parameters (e.g., task 123 264 completion times and error rate). Most methods require a large pool of test persons in order to create statistically viable results. In aviation, flight crews, engineers, and technicians are highly trained professionals with expertise in their specific domains. Their performance, appraisal, and feedback during experiments are more profound. Hence, experiments with these subject matter experts (SME), even with small sample sizes, return valuable feedback in particular on the operational implications of a proposed solution (Reuzeau and Nibbelke 2004). Figure 1 shows different prototypes and their level of fidelity, costs, and flexibility. A very low-fidelity prototype can be a paper sketch or a storyboard that is used to walk through sequences of events. If a concept gets more mature, stand-alone prototypes based on desktop computers with more or less complex simulations and interactions can be used. These kinds of evaluations are referred to as desktop research. Today, the systems of flight decks are highly interlinked and coupled (Wise et al. 2009). Therefore, an integrated evaluation in a rather holistic flight deck is essential. This simulator research can take place in engineering mock-ups, i.e., fixed based hardware flight simulators specially equipped for engineering needs or even high-fidelity certified motion-based full flight simulators. From using low-fidelity paper sketches to high-fidelity full flight simulators, the costs increase exponentially, whereas the level of flexibility, i.e., the ease of changing, integrating, and testing a prototype decreases. In order to transfer an HMI concept from desktop research to the simulator research domain, a huge additional effort in cost and time is necessary with the consequence of losing flexibility. For human factors engineering, this means that evaluations of HMI technologies can take place either in a flexible, low-fidelity desktop-based environment or in a high-fidelity hardware simulator with a product that is already in a rather mature state and is not subject to substantial changes anymore. Meister and Gawron (2009) analyzed human factors-related research papers and showed that approximately half of the studies, which did involve onsite testing with participants, took place in Fig. 1 Simulator continuum for human factors research 123 Cogn Tech Work (2017) 19:263–277 laboratory conditions or low-fidelity part-task simulators. The other half was conducted in full flight simulators or even in real flight conditions. The authors of the work presented here believe that there is a broad spectrum between these two options that, nevertheless, has mostly been ignored so far in the context of HF evaluations. This research aims to fill the gap in human factors engineering in flight deck design with the concept of immersive virtual worlds. With the help of virtual reality (VR), a subject can experience a three-dimensional space by wearing a head-mounted display (HMD) or 3D glasses with an attached head and finger tracking system. In this paper, a virtual cockpit mock-up that is connected to a flight simulation will be presented. The major contribution of this research is providing engineers with a tool to evaluate their ideas in a holistic operational environment already in an early stage of the design process. For these evaluations, human factors methods have been integrated in the system. With the specific modular architecture of the system, new cockpit concepts or single HMI components can be integrated easily, as can be additional human factors methods. This paper is structured as follows: In the next section, a brief overview of related work, in particular existing virtual reality cockpit simulators, will be provided. Afterward, the architecture of the virtual reality flight simulator and the means of visualization and interaction will be described. In ‘‘Human factors methodologies,’’ the integration of human factors methods will be discussed. Three user studies, which range from basic cognitive research to HMI evaluation and development, will be presented in ‘‘Experimental research.’’ These studies show the capabilities of this system for HF evaluations and the validity of the integrated HF methods. 2 Related work Virtual reality is already in use in the development of flight decks today. It is common practice for exploring ergonomic or visibility aspects of aircraft cockpits (Goutal Cogn Tech Work (2017) 19:263–277 2000). For ergonomic applications, the digital cockpit mock-up can be fully static without any functionality— similar to a wooden mock-up. To investigate cognitive aspects of an aircraft cockpit, an operational context is necessary though. A digital cockpit mock-up of a fixed-wing aircraft has been connected to a flight simulation for the purpose of pilot training by the Technical University of Darmstadt (Dörr et al. 2001; Dörr 2004). This system has been evaluated in regard of usability aspects (Hüsgen and Klingauf 2005) and was evaluated with airline pilots as a low-cost procedure trainer (Bauer and Klingauf 2008). There are similar systems for rotary-wing aircraft. The Airbus Helicopters Enhanced Virtual Environment (EVE) is a virtual cockpit that focuses on training (Bauer and Klingauf 2008) and is used in air crash investigations (Bauer 2014). The University of Istanbul also presented a low-cost virtual reality flight simulator for rotary-wing aircraft in 2009 (Yavrucuk et al. 2009). In contrast to existing systems, which mainly have a focus on training, the simulator presented in this paper focuses on the rapid prototyping, integration, and evaluation of HMI components. 265 based on ROS, each of these main components can be replaced, e.g., with a different type of tracking system or a different flight simulation software. Yet, one has to keep in mind that the usability of the system and the obtained results highly depend on the used hardware and software components. In the following section, the architecture with the commonly used components used up to now as well as the possible options of integrating additional components will be discussed. 3.1 The tracking system The optical tracking system delivers six degrees of freedom information of the user’s head and the position of his or her hands and each finger. These data are then transformed into the local coordinate system of the flight simulation. The transformed hand tracking data drive a geometry that resembles a human hand. The transformed head tracking data are connected to the virtual camera inside the three-dimensional cockpit in the flight simulation (Aslandere et al. 2014). In order to achieve stereoscopic vision, a separate module transforms the rendering output and performs the respective optical transformations for each eye (Southard 1993). 3 The virtual reality flight simulator 3.2 Visualization The virtual reality flight simulator that is presented in this paper consists of a virtual functional cockpit that is connected to commercially available flight simulation software. For experiencing this virtual mock-up, three distinct features are needed: a tracking system, an output device for visualization, and the possibility to interact with the flight simulation and the virtual mock-up (Dreyer et al. 2014). The modular architecture of the VRFS (see Fig. 2) is based on the robot operating system (ROS), an opensource network framework based on a publisher/subscriber system (Quigley et al. 2009). Using ROS has certain advantages over a custom-made solution or existing proprietary frameworks: Due to an active user base, ROS is constantly evolving, it has been ported to many devices and platforms, and it is well documented and supported. This leads to low entry barriers for developers, which is an important factor for the acceptance of a simulation environment. Every component in the system is connected to ROS and therefore has bidirectional access to information like telemetry data from the flight simulation or position data from the tracking system. The core system consists of basic hardware flight controls, a flight simulation (e.g., X-plane), the external tracking system, a head-mounted display (HMD), and a finger-tracking system. Due to the modular architecture The video data that are provided by the simulation can be presented to the subject by using a head-mounted display. The HMD used for this research has a resolution of 1280 9 1024 pixels per eye within a diagonal field of view (FOV) of 60 . Head-mounted displays with a higher FOV exist, but often the resolution does not increase accordingly. This leads to a smaller pixel density (i.e., pixel per degree FOV), which has a negative influence on the readability (e.g., symbols on displays) in the VR. If a higher FOV is necessary, a so-called powerwall can also be used in which the two images are displayed frame sequentially on a screen via a 3D-capable projector. A user sitting in front of the screen wearing motion-tracked shutter glasses gets an immersive stereoscopic picture of the simulation. 3.3 Prototype integration The intention of the modular and distributed approach of the VRFS is to make it convenient to integrate new software and hardware components like new system simulations or novel flight displays and their content. The robot operating system (ROS), which is used as a simulation framework, supports the transmission of numerical, audio, or video data and can be used independently of the 123 266 Cogn Tech Work (2017) 19:263–277 Fig. 2 Architecture of the virtual reality flight simulator operating system or programming language (Crick et al. 2011). For prototypes with new display content, a common use case, three levels of integration exist: (a) a static image or a playback of display content, (b) display content that is driven by simulation telemetry data, and (c) and bidirectional interactive display content (Oberhauser et al. 2016). These prototypes can be integrated without modifying the base system. A side effect of using ROS is the built-in ability to record the complete communication that happens in the system—a feature that is of particular value when it comes to post-analysis of a trial. 123 3.4 Virtual collaboration Modern flight crews consist of a pilot flying and a pilot non-flying with a strict allocation of tasks and responsibilities, the so-called crew resource management (Helmreich et al. 1999). This interaction has to be considered in the development and evaluation of new HMI components. Hence, it is preferable to perform a user trial with a twoperson crew. As all telemetry data are shared across the framework, it is easy to set up a second, identical instance of the flight simulation. The virtual camera of this second instance is driven by the head tracking system of the second pilot. In Cogn Tech Work (2017) 19:263–277 Fig. 3 Virtual collaboration in the virtual reality flight simulator order to be able to see each other’s head and fingers and interact with the second pilot, all tracked limbs are visualized in both instances as shown in Fig. 3. 267 Adding hardware controls to the digital mock-up is a balancing act between a fully flexible, cost-effective environment with no hardware elements and one with costly hardware elements but decreased flexibility. In an extreme case, this could lead to a full hardware mock-up, which undermines the purpose of a flexible VR environment. On the other hand, the product maturation process leads to an increasing product feature determination. By equipping the VRFS with this mature hardware, the simulation platform can grow with the project’s progress from a flexible rapid prototyping environment to a higher-fidelity engineering mock-up as shown in Fig. 5. Regardless at which point in the development process virtual reality is used to evaluate HMI components, the right human factors method must be chosen on a case-bycase basis. The following section describes the methodologies that have been used in the VRFS so far. 3.5 Interaction methods 4 Human factors methodologies Different methods have been implemented to give the user control over the simulation and to interact with cockpit elements. The interaction can be fully virtual as a collision detection system is implemented (see Fig. 4): If a collision between a virtual button geometry and a virtual finger geometry is detected, a command is sent to the simulation (Aslandere et al. 2014). In this case, the user has no haptic feedback, which makes the usability of these control elements challenging. With mechanisms to prevent dual-activation and by adapting the size of the collision volumes, the interaction with fully virtual elements can be enhanced (Aslandere et al. 2015; Hüsgen and Klingauf 2005). By placing plywood or acrylic glass plates at the spatial position of virtual buttons, a simple haptic feedback can be generated. With this method, the usability of the virtual elements can be further enhanced. Interactive hardware elements (e.g., controls and/or actuators) can be connected to the simulation as well. If those elements are placed in the exact same spatial position as in the digital mock-up, a socalled mixed mock-up is created. Frequently used levers and rotary buttons should be provided as hardware elements for a fast and intuitive use, as illustrated in Fig. 4. In aviation, workload (WL) and situation awareness (SA) are the two central concepts when it comes to the evaluation of flight decks (Wickens 2002). In the VRFS, several human factors methodologies have been applied to measure these concepts in multiple ways. These methods can be divided into subjective methods, i.e., self-assessment information provided verbally or in written form by the pilot, and objective methods, i.e., data gathered from independent sources. Another commonly used human factors metric is the distribution of visual attention. The focus of attention can be shifted overt, i.e., an eye movement, or covert, i.e., a shift of attention without an eye movement (Posner 1980; Hunt and Kingstone 2003). A shift of visual attention to a certain element does not necessarily involve its perception or even comprehension (Just and Carpenter 1980). Yet, visual attention is an important factor when it comes to the interface design of head-up displays (HUD) and glass cockpit screens. The HF methods for assessing the visual attention, both overt and covert, as well as methods for assessing workload Fig. 4 Mixed mock-up and full virtual interaction 123 268 Cogn Tech Work (2017) 19:263–277 Fig. 5 Mixed mock-up in the cockpit development process and situation awareness and their application in the virtual environment are presented in the following sections. 4.1 Eye tracking In order to evaluate the pilot’’s overt information-gathering process, an eye tracking analysis is necessary. This is of particular interest for the evaluation of displays like primary flight displays (PFD) or head-up displays (Jorna and Hoogeboom 2004). In such systems, the eye movement provides indications on the pilot’s visual perception. With the number of fixations, glances, and transitions in combination with other data, like physiological information, conclusions about the pilot’s workload can be drawn. Thus, an eye tracking system is useful for many human factors evaluations of modern flight decks and is therefore integrated in the VRFS. In this context, Jorna and Hoogeboom (2004) state: ‘‘The use of eye based data is attractive as it reveals the strategies that are being used by the flight crew to access the visual data available on the flight deck.’’ Optical eye tracking systems need a clear line of sight to the subject’s eye. In a VR environment, this view is obstructed by an HMD or by 3D glasses. To gather eye tracking data when using an HMD, a small camera is placed close to the pilot’s eye. In order to adapt to different subjects, this camera is attached to an adjustable ring around the HMD’s eye piece to get a clear view of the pilot’s eye. Tests showed that this system is reliable and capable of adjusting to most test persons even when disturbed by glasses or makeup (Liesecke 2013). The system merges the eye camera image with an image from a field camera and calculates the eye gaze based on a onetime calibration. In the VRFS, this field camera is substituted 123 by the visual data that is rendered by the flight simulation. For this purpose, a synchronized instance of the flight simulation is created. This additional view is used solely for the eye tracking system and can be enriched with additional information like simulation time or eye tracking markers as illustrated in Fig. 6. With these markers, areas of interest (AOI) can be defined in post–processing, and an automated evaluation of eye fixations and transitions is possible. Another way of processing eye tracking data is to transform the 2D information of the eye gaze into threedimensional space. In combination with the three-dimensional eye gaze vector, the data from the head tracking system, and a virtual mock-up of the environment, it is possible to calculate the ‘‘collision’’ of the eye gaze at every point of time. Based on this information, a heat map visualization of the eye gaze on the three-dimensional cockpit geometry can be created as shown in Fig. 7(left). If the virtual cockpit is annotated with AOIs, as illustrated in Fig. 7(right), further quantitative analysis of the eye gaze movement can be conducted automatically or semiautomatically. 4.2 Motion analysis The tracking system delivers data about the head and hand movements and is one of the key components to create the virtual reality environment. The data can be used to draw conclusions on the physical workload. The frequency and amplitude of head and/or hand movements can be used as a measure of the amount of physical workload. Another measure can be the dwell time for ergonomically unfavorable positions like overhead operations. Whether the analysis of motion data has an added value has to be Cogn Tech Work (2017) 19:263–277 269 Fig. 6 Eye tracking view with virtual markers and areas of interest Fig. 7 Three-dimensional eye tracking heat map and areas of interest decided on a case-by-case basis according to the specific HF questions at hand. 4.3 Physiological data A parameter that can easily be measured is the heart rate. Among others, this information can reflect psychological concepts like ‘‘mental load’’ and ‘‘effort’’ (Jorna and Hoogeboom 2004). In the VRFS, these data are collected by using a pulse sensor with a sensor attached to the pilot’s ear. This sensor is also connected to the simulation framework and can therefore be recorded. Thus, the heart rate can be correlated with the respective flight phases. 4.4 ATTENDO The ATTENDO method is used for assessing the distribution of visual attention. It is based on the secondary task paradigm, i.e., measuring the reaction of a visual stimulus by the pilot (Hillebrand 2013). The visual targets are small rectangles or circles and are pseudo-randomly displayed spatially and temporarily in the subject’s field of view for a short period of time (e.g., 200 ms). The primary task, e.g., flying the aircraft, has to be executed and monitored by the pilot permanently. Whenever a target is visible for the pilot, he or she should react by pressing a button. If the pilot reacts more often to targets in a certain area, this area has a higher degree of visual attention. The method has been used to evaluate the distribution of visual attention of different head-up displays, showing visual tunneling effects in highly de-cluttered and centered HUD symbologies (Hillebrand et al. 2012). ATTENDO can be used to evaluate the distribution of visual attention in a two-dimensional plane as well as in three-dimensional space (Michalczik et al. 2013). The ATTENDO method has been integrated into the VRFS using a plugin for the flight simulation that can be used in the virtual reality flight simulator as well as on a stand-alone desktop computer. The targets have been recreated as three-dimensional objects. They can be displayed inside as well as outside of the cockpit. The position, appearance time, and the properties of the objects are controlled by a sequential configuration file. 4.5 Questionnaires The NASA Task Load Index (NASA-TLX) is a wellknown multi-dimensional subjective method for assessing 123 270 Cogn Tech Work (2017) 19:263–277 the workload of a task (Hart and Staveland 1988; Hart 2006). This method can be applied with a simple questionnaire after a task is completed in the VR. In order not to interrupt the immersive virtual reality experience, i.e., removing the HMD or glasses, this questionnaire can also be displayed inside a virtual environment (Dreyer and Hillebrand 2010). The situation awareness rating technique (SART) can be used to assess the situation awareness after a task in the VR has been completed. It is suitable for short experiment sessions (Selcon and Taylor 1990). Similar to the NASATLX, this subjective method can be conducted with or without VR equipment. 5 Experimental research With the help of the virtual reality flight simulator and the integrated human factors methods, several user studies have been conducted. In this section, three of these studies will be presented (for an overview, please see Table 1). The studies show the application of the presented methods and represent a wide spectrum of technology maturity and complexity of scenarios. The types of studies, according to Deaton and Morrison (2009), range from basic research for testing a new theoretical concept, to applied research with a software demonstrator, to advanced development for evaluating a hardware prototype. 5.1 Advanced head-up display evaluation 5.1.1 Introduction Head-up displays present information in the pilot s head-up field of view. These displays have three design goals: (1) reducing the amount of visual scanning between the information in the cockpit and the outside world, (2) reducing re-accommodation between the outside and the inside head-down view, and (3) presenting additional augmented information about the outside world (Wickens et al. 2004). Head-up guidance systems (HUGS) provide additional information on the optimal flight path and energy management, enabling a pilot to stay head-up in all phases of flight and improving situation awareness. Nevertheless, when using such a system in a standard instrument landing system (ILS) approach, according to procedures, a pilot has to gather information regarding the final checklist items which are ‘‘Landing Gear,’’ ‘‘Thrust Reversers’’ and ‘‘Flaps’’ from head-down displays. In consequence, he or she needs to re-accommodate to the outside world (Bandow 2006). As part of the ALICIA (All Condition Operations Infrastructure) project, an advanced head-up display symbology was developed that eradicates the procedural need for head-down information gathering in a standard ILS approach (Dreyer et al. 2014). This HUD has a single indication, a simple green disk for all system states that have to be checked in this phase of the approach. 5.1.2 Methodology This so-called Green Disc Concept was integrated into a conventional head-up display. To exclude confounding factors stemming from the VR environment, a comparison of this novel HUD with a conventional one was chosen, both evaluated in the virtual environment. The scenario for both experiments was a low-visibility instrument landing system (ILS) approach to the airport of Clermont–Ferrand (LFLC). This means that an electronic navigation aid is present and guides the pilot to the airport; visual contact with the runway has to be established at least 100 ft above Table 1 Selection of studies performed with the virtual reality flight simulator Study Type of study Technology Scenario Human factors methods (1) Advanced HUD evaluations Applied research Virtual head-up display demonstrator Realistic low-visibility approach scenario Eye tracking analysis NASA-TLX SART Heart rate analysis (2) Peripheral horizon display evaluation Basic research Early research prototype Simplified, non-operational flying task ATTENDO (3) System management study Advanced development Hardware prototype Complex realistic multi-system failure scenario Head tracking analysis NASA-TLX SART Temporal task performance 123 Cogn Tech Work (2017) 19:263–277 ground. Starting with the initial approach fix, eventually this approach ends in a go-around, a procedure to abort an approach and regain altitude, as visual contact with the runway cannot be established. The experiment supervisor took the role of the co-pilot to ensure conformity with operational procedures. Eleven pilots from different German airlines participated in the trials. In order to assess workload and situation awareness, a SART and a NASATLX questionnaire had to be completed after each approach. Head movements and eye gaze, acquired through the built-in eye tracking system, were used to distinguish between head-up and head-down phases. 5.1.3 Participants Eleven male commercial airline pilots from several German airlines participated in the trials with a mean age of 49 years and an average experience of 10,200 flight hours. 5.1.4 Results The results show that the subjectively measured workload decreased. At the same time, the subjectively measured situation awareness for the landing task increased (Dreyer et al. 2014). This can be attributed to the fact that the headup percentage was increased significantly due to the introduced ‘‘Green Disc Concept’’ as illustrated in Fig. 8. A similar scenario, yet using a conventional head-up guidance system, was used in the research conducted by Bandow (2006). For the presented user study in the VRFS, the same operational procedures were applied as well as a similar conventional HUD. Figure 8 shows that the average head-up percentage in the full flight simulator study (86.70%) is similar to the conventional scenario in the virtual reality flight simulator (87.55%). This indicates that the operational behavior in virtual reality is comparable to the full flight simulator environment. 271 Regarding physiological data, the heart rate was captured in both user studies. Figure 9 shows an illustration of the average heart rate across all subjects. The data is normalized based on the resting heart rate (Bokranz and Landau 1991). The time axis is shifted to align with the goaround event and covers a range of 5 min before and 30 s after the event. The heart rate in the full flight simulator is substantially higher than in the VRFS, as well as the increase shortly before the go-around. This increase, albeit at a much lower level, is also visible in the heart rate data of the virtual reality scenario. With a section of the heart rate gathered in the virtual reality trials, a discontinuity regression analysis (Shadish et al. 2002) was conducted as illustrated in Fig. 9. This method shows a discontinuity of the data 30 s before the go-around. This leads to the hypothesis that this increase of heart rate is significant and triggered by the go-around event. Further research is necessary to investigate this effect. Figure 10 shows an eye tracking heat map of a head-up display in primary mode and in the de-cluttered A3 mode. In this study the T-scan pattern, which is typical for pilots trained in using head-up displays, could clearly be observed in the eye tracking analysis for the primary mode. When switching to the A3 mode, this wide scanning pattern is replaced by a shift of attention to the center of the display as expected (Bandow 2006). The next study deals with the so-called mental tunneling effect that stems from this behavior. 5.2 Peripheral horizon display evaluation 5.2.1 Introduction In the final approach phase, head-up displays equipped with HUGS get highly de-cluttered. This means that speed and altitude tapes disappear and most of the information is Fig. 8 Head-up percentages during a low-visibility CATIIIA approach 123 272 Cogn Tech Work (2017) 19:263–277 Fig. 9 Normalized heart rates in the VRFS and the full flight simulator study and a regression discontinuity analysis Fig. 10 Head-up display and eye-scanning patterns moved to the center of the display. This shifts the pilot’s attention to the runway centerline and the central flight guidance symbology. As a side effect, a mental or cognitive tunneling effect may occur (Hillebrand 2012; Bandow 2006). To counter this effect, Bandow (2006) proposed a peripheral vision horizon display (PVHD) that is based on the Malcolm Horizon (Malcolm 1984): This system consists of two artificial horizons that are integrated into the struts of the cockpit as shown in Fig. 11. This peripheral stimulus should expand or even eliminate the mental tunnel created by the de-cluttered and centered HUD/HUGS symbology. Fig. 11 Peripheral horizon display 5.2.2 Methodology As in this study the influence of the peripheral stimulus of the PVHD is the subject of research, a high field of view is necessary. The output device powerwall offers this high field of view. The pilot is placed right in front of the screen with the two struts on the far left and far right of his field of view. To evaluate the distribution of visual attention, the ATTENDO method is used. With the stereoscopic vision 123 enabled by the shutter glasses, the visual attention is also measured in two separate depth layers. A comparable flying task with defined roll and pitch values is pre-programmed. The pilot has to follow these roll and pitch commands as exactly as possible while maintaining the airspeed. Simultaneously, the ATTENDO targets are displayed randomly and a reaction using the fire button on the joystick is required. Cogn Tech Work (2017) 19:263–277 273 5.2.3 Participants As the flying task in this scenario was simplified, no operational flying skills were required in order to participate. Thus, in this study, 11 licensed pilots and 12 nonpilots participated. 5.2.4 Results The results indicate no significant difference in the distribution of visual attention between the flights with and the flights without peripheral horizon display. Despite this negative result, further studies on the influence of the PVHD should be conducted to rule out confounding factors stemming from the experimental setup and the virtual environment. In particular, an experiment with licensed commercial pilots and a peripheral vision HMD should be undertaken for that purpose. Yet, the trials show that the ATTENDO method itself is feasible in virtual reality environments. 5.3 System management study Another user study unrelated to head-up displays aims at simplifying the ‘‘Manage Systems’’ task in the flight deck. Most of the aircraft systems are controlled via the overhead panel (OHP). In consequence, changes in aircraft systems are directly reflected in the design of this panel. In this consecutive study, a new OHP was designed and accompanied by a system management touch screen device. The goal of this study is to evaluate the impact on the human performance when conducting system management tasks using this new HMI. The requirements for the system management controls stem from the comparison of existing aircraft systems, their HMI, and the development of a hypothetical generic aircraft, including simplified HMI concepts, further system automation, and a transfer of system management functions to the head-down area. Fig. 12 Virtual cockpit with the novel overhead panel and the softICP beforehand. The device is placed at a matching spatial position, connected to the simulator’s telemetry data using the capabilities of the robot operating system and its screen content is transferred into the virtual cockpit. Similar to study (1), a comparative approach has been chosen for both trials. Here, the system management procedures and the OHP of an Airbus A320 are compared to the novel system management approach and the respective OHP. In the first trial, two malfunctions that are followed by rather complex system management tasks have been chosen as scenarios. The first malfunction, a bleed air fault, occurs in cruise flight. This forces the pilot to descend. Shortly after reaching the new altitude level, an engine fire occurs, followed by further system management tasks. In the second trial, a scenario with multiple system failures that result in a fuel leak was chosen. With the available system management controls, the pilot has to locate and isolate the leak (Dreyer and Oberhauser 2016). During the scenarios, eye or head tracking data, the heart rate, telemetry data and finger trackin... Purchase answer to see full attachment