The V-22 Osprey is the world’s first production tiltrotor aircraft. It’s a complex machine, challenging to maintain. How do technicians master the nuances of a craft with the takeoff of a helicopter, speed and range of an airplane, storage of a transport, massive rotors that fold, and wings that rotate? Immersive simulation.

Raytheon Intelligence & Space, a Raytheon Technologies business, has teamed with serious games developer Virtual Heroes to give the U.S. Naval Air Systems Command a virtual Osprey maintenance-training system. The trainer simulates realistic maintenance scenarios via a gaming engine and a portable three-screen touch-sensitive gaming computer. This innovative approach minimizes the bulk and cost of current training hardware.

“As the modern battlespace evolves, the way the military trains must evolve with it,” said Dean Hoover, Systems Engineering section manager for Raytheon Intelligence & Space’s Virtual Systems division. “Adapting the latest technology can be the key differentiator in whether service members are positioned for military victory.”

 

Evolution of Osprey Training

The Bell Boeing V-22 Osprey is a joint-service, multi-mission aircraft that can perform vertical take-off and landing. It carries out VTOL missions as effectively as a conventional helicopter while also having the long-range cruise abilities of a twin turboprop aircraft. The U.S. Marine Corps, Navy, and Air Force use the Osprey to transport troops, equipment, and supplies, as well as conduct combat search and rescue and long-range special operations missions.

“Any training system that teaches maintenance and repair of the Osprey must capture the complexities of such a unique aircraft,” said Randy Brown, vice president and manager of the Virtual Heroes Division of ARA. “Being able to accurately visualize spatial elements of this environment is key, since various activities can often only be performed from specific locations on or around an airframe.”

The evolution of the V-22 virtual trainer is rooted in Raytheon Intelligence & Space’s 30-plus years of experience working on the Osprey through its Software Support Activity (SSA) in Indianapolis. Raytheon Intelligence & Space collaborated with a team lead by Bell and Boeing to develop and test the aircraft Operational Flight Program—the code that controls all the avionics systems on the aircraft, and the Multi-Function Display code—the program that builds the displays the pilots interact with to monitor and control the aircraft.

In 2001, the SSA began work on two large-scale maintenance trainers called the Avionics Functional V-22 Trainer, AFVT, and the Trainer Fault Insertion System, TFIS. The AFVT is a cockpit-sized trainer consisting of nine touch screens displaying 3D graphical touch interface representations of all cockpit controls. The TFIS used a stricken airframe repurposed to simulate hardware faults in V-22 training scenarios.

 

 

At the time, these systems were state of the art; however, they became difficult and costly to maintain as the years went on. Further, they were limited by older graphics technology and the fact that only a small number of students could train on these systems at a time.

Fast forward to 2015. The Naval Air Systems Command, NAVAIR, asked Raytheon Intelligence & Space to research what it would take to create a portable version of the AFVT system. They were looking for something that could eventually replace the TFIS, too.

Their original idea was to take the legacy software from the AFVT and run it on a portable system. This seemed simple, but “the GLStudio graphics engine used on the large AFVT system was very processor intensive, not very realistic looking and also costly,” said Hoover. A Raytheon Intelligence & Space engineer came up with the idea of rewriting the graphics portion of the system using Epic Games’ Unreal Engine. Said Hoover, “This gaming engine was becoming very popular in the gaming community; it’s the same engine used to build the massive online multiplayer game Fortnite.”

Raytheon Intelligence & Space pitched this idea to NAVAIR, and by mid-2015 they had a contract to create the first demonstration prototype. Looking for assistance in developing with the Unreal Engine, Raytheon Intelligence & Space looked to companies in the Department of Defense. They came upon defense contractor Virtual Heroes, serious games experts with over 15 years’ experience leveraging Unreal to create 3D training, simulation, and virtual-world experiences. Shortly after Hoover and Brown met, collaboration began on the next generation of AFVT: PAFVT, the Portable Aircraft Functional V-22 Trainer. 

 

 

Virtual, Portable Training

Hoover and Brown aimed to create a virtual system that could run on any standard gaming-capable PC and be easily transported anywhere training was needed. They envisioned a 3D, immersive environment typical of modern video games.

The advantages of a virtual environment were clear. Brown said, “Training in a virtual environment eliminates the need to maintain hardware-based, cockpit-sized simulators. It also provides improved context and reference to training activities, versus text-based, multiple-choice, or 2D approaches.”

Brown added, “By rendering the full 3D environment in a real-time, high-fidelity game engine technology like the Unreal Engine 4, the learner has the ability to understand not only the required steps for various procedures, but also the relative locations or positions they must move to for successful task completion.” This avoids artificial transitions or cuts from one task to the next, which could cause the learner to lose the context of the training scenario.

Hoover said, “The key to success here is our integrating gaming technology, immersive simulations, and real avionics software. This allows for hyper-realistic immersive interactions with the aircraft.”

In the virtual environment, modeled controls and services interact with a version of the actual avionics software that runs on the V-22. Raytheon Intelligence & Space is the developer of the V-22 Display Engine (VDE) software; VDE emulates a selection of V-22 Cockpit Management System (CMS) capabilities, including those presented on the aircraft’s Multi-Function Displays.

Hoover said, “These VDE displays are used in the development and testing of the V-22, allowing for training with an exact duplicate of the display systems the crews will see on the aircraft. These unique tech integrations eliminate negative training elements of other systems that do not exactly duplicate the real environment.” By building on the behavioral software systems and training sequences Raytheon Intelligence & Space had already created for the V-22, Virtual Heroes could map very high-fidelity user interface elements to in-world system control interfaces.

One key challenge in building the Osprey in the Unreal Engine was the sheer number of control surfaces to program, or “buttonology.”

Interface elements include hundreds of panels, buttons, knobs, ports, and levers inside and outside the aircraft. The team ended up creating a mapping system that enabled Virtual Heroes artists to quickly model and encode each interactive element without performing unique time-consuming manual steps for each one. (Manual steps include, for example, assigning element IDs to physical 3D button models and then encoding every potential state of on, off, pressed, released, lit, unlit, etc. for each control.) This automation greatly accelerated the creation of assets and their linkage to Raytheon’s system behavioral models.

In the end, the team created a learning experience that’s truly immersive. When equipped with a VR headset and controllers, the learner can look and move around, locate and interact with controls, and practice actual maintenance procedures. These procedures involve diagnosing problems such as improper installation, apparent defects, and unsatisfactory operation. Two examples of procedures are engine startup and blade fold wing stow. The latter is a complex procedure that trainees don’t often get to practice in a real aircraft, based on the real potential to damage the aircraft if not done correctly. Using the virtual trainer, trainees can practice over and over until they get it right.

The current virtual trainer prototype runs on a three-monitor portable computer, making it mobile and accessible. Says Hoover, “Raytheon Intelligence & Space is working with V-22 customers to deploy these portable training devices outside the training rooms existing today. In the field, squadrons can use these devices to maintain skill proficiency.”

 

 

Next-steps in VR Training

Raytheon Intelligence & Space and Virtual Heroes plan to add more functionality to PAFVT in the future. They will expand training procedures to include hydraulics, avionics, and other relevant systems, and broaden the capability to different aircrafts entirely.

For a true VR experience, the Raytheon/Virtual Heroes team has architected the virtual V-22 environment so it can run on a fully-immersive 3D virtual head-mounted display. While this is not part of the currently contracted system, the team is making sure this capability is ready to go if and when the need arises. Said Brown, “With VR, the learner experiences a greatly increased opportunity to focus on the tasks at hand without distractions. This is especially valuable within room-scale learning environments such as the cockpit of the Osprey.”

The team are also working together on other VR training systems for the Department of Defense, including tactical operations and offensive weapons trainers. Says, Hoover, “DoD training needs lend themselves very directly to the kind of immersive, engaging virtual environments we want to create.”

Adds Brown, “We’ll continue to blend virtual and live training environments through intelligent combinations of VR and AR technologies. This will keep us busy solving challenging and interesting problems for the Warfighter for many years to come.”

As aviation technology evolves, so will training systems. Virtual simulations are becoming more and more realistic, with virtual-reality enabling users to experience a close facsimile of the real world. Companies who stay on top of these advances, such as Raytheon Intelligence & Space and the Virtual Heroes Division of ARA, are providing the warfighter with better training tools and helping to ensure mission success.

---

This article was originally posted on Jun 14, 2021