VR poses challenges for software testing
You know the scene. You’re walking down the street in a town, maybe your hometown, on a quiet Tuesday afternoon. The sun is shining, people are sitting outside a coffee shop chatting and browsing, and you can hear birds chirping and the drone of distant traffic.
You turn the corner into a side street, and suddenly an enormous slime-covered viscous alien blob is looming in front of you. Its eyes are as big as dinner plates and the rippling bristles on its face seem to change color several times.
You can see the drool on its lower lip as one of its weird spider-like limbs reaches for a stun-ray from its shoulder holster. You reach for your own laser. Then things… start… to… slow… down.
But it’s not your combat training that’s making time slow down: it’s a bug. You remove your virtual reality headset and fire up the bug tracking database.
As you probably know, testing VR applications presents its own challenges, but also huge opportunities. Virtual reality and its virtual cousin augmented reality are big business nowadays. For example, Microsoft recently announced a deal worth approximately $500 million with the U.S. Army to customize development for its HoloLens headset.
And as the cost of the hardware continues to drop, headsets could eventually be as integral to gaming consoles in the home as motion sensors were a few years ago. So what are the most important things to know about testing VR software?
VR software is still software, so the usual principles of testing still apply. For example, if it crashes, it’s a bug. If it deviates from the specification, it’s a bug. If the specification says the bug-eyed alien should be blue but it looks more like yellow, then it’s a bug.
Though you can be sure the developer will claim that the yellowness of aliens is a feature and not a bug.
And because most VR applications are at least broadly similar to video games, you can expect to find the same sort of bugs that are relatively common in video games, such as visual glitches, sound errors, and controller issues. So you could be speeding across the open waves on your jet-ski, you look up to see the cloudless sky, you look down—and you see just the handlebars and the water being smoothly shorn beneath your feet, and the rest of jet-ski has disappeared! That would be a certain bug!
Where VR testing differs most from testing 2D games is that it’s immersive. This means that the user has to be in the app or game, which presents the challenge that the tester has little or no communication with the outside world while testing.
For one thing, unless testers have Jedi-like coordination and typing skills, they’re not going to be able to write up a bug report while wearing a headset. Similarly, they cannot read a checklist of points to be tested while they are wearing their headsets. So they will frequently have to switch between the headset and a regular computer screen, and they probably won’t be as efficient as if they were testing a regular game or app.
From the tester’s perspective, this also means that good observation, memorization, and recollection skills are great assets to bring to the game. If you can keep a checklist of bullet points in your head before you put on your headset, you can expect to get better results than if you just jump unprepared into the virtual world.
Similarly, the more detail the tester notices in the game, the better. Did you look around to see the birds that were chirping? Were they pecking at breadcrumbs outside the coffee shop? Could you see the distant highway with the droning traffic? What sort of sound did the alien make?
Another big issue to consider is what is known as “VR sickness.” This is typically characterized by symptoms similar to those of sea-sickness: dizziness, nausea, and disorientation. Oculus, one of the biggest names in VR, refers to this in its Health and Safety manual.
As this is a health and safety issue, it should be taken very seriously. Failure to follow best practices could lead to adverse consequences for your company, such as employee dissatisfaction, absenteeism, or even litigation.
While the reasons for VR sickness are not fully understood, there are some factors with which it is known to be correlated. For example, low framerate and low-resolution graphics are both associated with VR sickness. So ensuring that your test rig is powerful enough to provide an adequate level of performance is a necessity.
Another known factor for moderating VR sickness is sitting rather than standing while playing, so comfortable seating for the test team is another must-have. Also, some people are simply more susceptible to motion sickness than others, so picking an appropriate test team is important.
There are also design factors that VR developers should be aware of that ameliorate the symptoms. These include avoiding certain types of motion, such as climbing stairs in the virtual environment and reducing the field of view. However, a side-effect can be a reduction in how immersive the game feels.
Taking regular breaks from the headset is also recommended for testers. This might seem to be fairly standard advice, not just from the perspective of minimizing VR sickness, but also for general health and safety. You have to bear in mind that, for some people, playing games can be an addictive behavior, and we’ve all heard of an unfortunate gamer who died after playing a regular PC game for 22 hours straight.
VR games and their effects on user health are still a somewhat undiscovered country, so even if a tester insists that he or she feels fine and could easily keep testing, there should be processes in place to ensure that breaks are taken at least every couple of hours and that nobody tests more than a reasonable number of hours per day.
Testers should also feel empowered to report if they feel ill from a particular level or activity in the game and this feedback should go back to the game designers. Jumping from a fast-moving jet-ski onto to another might seem like a cool idea, but only if your customers have sea-legs like James Bond.
The future of reality
Augmented reality applications have a lot in common with virtual reality, so the above principles for testing should also be applied. However, recent studies of the latest AR technology have shown relatively low levels of motion sickness, and in fact, AR is poised to outgrow VR over the next few years, and their combined market is estimated to be over $90 billion dollars within five years. Which means the opportunities for developing and testing these technologies will also be growing.
In addition, new technological innovations that improve the VR experience are appearing all the time. For example, Otolith Labs, of Washington DC, is developing a prototype device that sits behind the VR user’s ear. The device sends white noise to the user’s vestibular system, which is the system responsible for balance and coordination, which blocks the root cause of motion sickness. So the future looks bright for these new realities.
That is, it all looks bright, until one quiet Tuesday afternoon, when you’re walking past your local coffee shop, people are sitting outside eating pierogi and sparrows are pecking at the pavement, and you hear what sounds like slime bubbles oozing, just before you turn the corner into a side street.