Does your VR training really work?

Virtual reality use cases have greatly matured over the past few years. VR has gone from an experimental technology to something that’s transforming the way enterprises work.

A large portion of the industry has moved past proof of concept into real world applications. A Grid Raster survey this spring found that 56 percent of executives said they had already implemented virtual or augmented reality, and 27 percent had fully deployed their solution and were looking to scale it up.

Training in particular has been one of the most successful use cases for virtual reality. PricewaterhouseCoopers found VR training was four times faster than classroom learning, Boeing uses VR to train their astronauts for space missions, and Volkswagen reported that training times were reduced by 50 percent using VR.

Boeing Starliner. (Image courtesy Varjo.)

VR makes learning from PowerPoint slides and videos look outdated in comparison.

But once the initial proof of concept is successful, how do you show that the training program is still effective after it is scaled up?

This is where a lot of companies struggle because they don’t have a way to scale assessment and prove out value throughout the organization.

In this article I’ll highlight our experience working with a medium-sized power utility company in the US, the challenges they faced and how they overcame them.

The utility company was looking for a better way to prepare employees for the dangers of the job and turned to VR.

(Image courtesy Master Lock.)

For the initial proof of concept, they designed a safety compliance scenario for Lock Out, Tag Out, a safety procedure used in industry and research settings to ensure that dangerous machines are properly shut off and not able to be started up again prior to the completion of maintenance or repair work.

Qualitatively, the new VR training program was a success. The employees were engaged and loved how realistic the VR experience felt.

However, issues arose when the training managers wanted to dig deeper into the assessment. During the sessions, they noticed some of the employees did extra steps for safety that weren’t included in the baseline assessment. They wanted to include these steps to the baseline and see how compliant employees were — but they couldn’t add the steps without reshipping a new build — building and distributing a new VR training — because they hard-coded assessments directly into the VR experience.

Then, a funnel was created through Unity Analytics to evaluate how many employees completed the original baseline steps. The specific assessment was based on the VR experience they designed, and this approach made it difficult to change their assessments based on new information or requirements. Completion was only qualified based on previously established criteria.

Management realized this wasn’t scalable. They didn’t want to ship a new build every time they tested something new or asked an additional question. VR training is an iterative process and a flexible system was necessary to proceed past the exploratory phase. They needed a solution that was robust enough to be scaled throughout the organization.

A new approach to assessment in VR training

The utility company took a new approach towards assessment. Instead of predetermining the outcomes, they looked holistically at the behavior and captured data. Using sensors from the VR headset and biometrics, they captured eye tracking, 3D positioning and controller actions.

Training managers were able to see exactly when something was interacted with or seen, and 3D environments like VR also gave them the ability to automatically subscribe metrics to objects.

For example, one of the training steps involved connecting the “Tag Lock” to the breaker box. During training, some employees took longer than others to locate it. With the new approach, the training managers could see exactly how long it took employees to find it. Data was collected for every stimuli, tool or hazard in the VR training experience.

By collecting the data beforehand and processing assessments in the cloud, edits could be made to the assessment whenever they needed to based on different new information. This opened up new possibilities because they were able to iterate much quicker.

There was no need to reship the experience every time they learned something new. Instead, they could retroactively evaluate behavior when they had more questions. This alternative approach provided the scalability they needed to iterate faster and more effectively.

But once they established a better system for measurement, they needed somewhere to store and keep track of all the data.

The utility company already had substantial training programs to ensure their workforce was trained and retrained at the appropriate cadence. They used a learning management system to manage, track and distribute all the training activities to the workforce. It was a major investment and included a large number of traditional training activities, such as video-based courses, audio, manuals, and tests.

VR became a portion of their overall learning and development program and needed to be integrated to the learning management system through an experience application programming interface. This is an industry API standard for activity providers to transmit results to a learning management system, and they turned to this standard to transmit employee performance in their VR training into the learning management system, which then tracks and understands the employees’ progress.

Albert Liu
Latest posts by Albert Liu (see all)