Designing in 3D immersive environments involves understanding how the whole body experiences an interaction. It’s all about knowing how the world really functions and deciding whether a high fidelity experience mimicking real life things, such as physics, is necessary to create a rewarding experience.
Creating a valuable experience in a virtual world involves understanding the human field of view, range of motion, viewing zones and avoiding perceptual cues that might cause simulator sickness. Luckily my PhD (many moons ago) was all about researching the level of fidelity required for the first ever bus driver training simulator in the UK.
Insights
My brief was to “Design 3D Symbology for Fast Jet Pilot Training”.
The first step was to understand how fast jet pilots make decisions, their skills, experience and the cues they rely on and what helps or hinders them. I read extensively and found a gem of a book called Pilot Stress from the 90s that explained situation awareness and cognitive overload. Then I went on to recruit two pilots to take part in semi-structured interviews.
Conducting user research with such a limited sample size might raise concerns about the generalisability of the findings. However, I selected highly experienced pilots with diverse backgrounds and focussed on gathering rich, qualitative data to ensure that I gained valuable insights. Each pilot constructed timelines of intense, real-life critical situations they’d encountered in combat and I used probes, based loosely on the critical decision method, to deep dive into the sensory, cognitive and environmental cues that the pilots used to successfully navigate and complete their missions.
I learned that maintaining a tactical advantage requires an exceptional level of situational awareness, the ability to process vast amounts of information rapidly, effectively assessing the environment, threats, and friendly forces and anticipating and reacting swiftly to changing circumstances.
So, my goal as a designer was to create 3D symbology that conveyed essential information without overwhelming the pilots with visual clutter.
3D Symbology Design Challenges
I had to ensure that pilots could quickly and accurately recognize and interpret the 3D cues in a rapidly changing environment so they had to be simple and legible. I had to include the following cues to help pilots assess the situation:
- Head-Up Display (HUD) Elements included flight parameters (airspeed, altitude), heading, navigation waypoints, weapon system status, and target tracking indicators directly in the pilot’s field of view.
- Threat Indicators, depicted as 3D symbols on the pilot’s display, allowing for quick identification and situational awareness. Ground-based threats included range of detection and effect, air-based threats such as enemy aircraft or missiles included relative position, altitude, and speed indicators.
- Terrain Awareness and Warning System (TAWS) included terrain contours, ground proximity warnings, and altitude callouts to alert pilots of potential hazards during low-altitude flight and approaches to the landing zone.
- Flight Path Vector (FPV) represents the projected path of the aircraft based on its current trajectory and flight parameters to help pilots to maintain their desired flight paths.
- Landing Aids included 3D representations of glide slopes, runway centerlines and aircraft position markers, assisting pilots in maintaining proper alignment and descent rate during approach and touchdown.
- Air Traffic Control (ATC) Communication Indicators included visual representations of the relative position, altitude, and speed of hostile aircraft to help pilots maintain situational awareness.
- System Status Indicators represented engine performance, fuel level and weapon system readiness to allow pilots to quickly assess the operational status of critical components during their flight.
- Formation Flying Cues to help pilots maintain formation position and spacing and to avoid collisions using virtual flight paths, relative position indicators, and visual cues for maintaining wingtip separation.
Complicated huh! I had to keep reminding myself that these pilots were the cream of the crop, were dedicated and committed to continuously improving their knowledge and skills and spent years completing specialist training and operating fast jet platforms.
Successful Outcomes
The pilots gave feedback on early 2D representations, then 3D representations on a 2D screen, then 3D representations using oculus rift.
Observing the pilots’ reactions and gathering feedback iteratively at each stage of the design process helped refine the symbology by prioritising the critical information and eliminating non-essential elements.
The outcome was clear symbology that pilots found easy to identify and interpret that could be tested in the dynamic environment of a full mission simulator.
