Augmented reality is moving us to the next frontier of task management

Until now, astronauts relied primarily on written and voice instructions when performing complex repair tasks or experiments in space. But Microsoft and NASA are working to change all that; a project called “Sidekick” uses Microsoft HoloLens to provide virtual aid to astronauts working in space. The goal of Sidekick is to meet station crews with assistance when and where they need it. The capability could lessen the amount of training that future crews will require and could be an invaluable resource for missions deep into our solar system, where communication delays complicate difficult operations.

The Sidekick project is part of a larger partnership formed by NASA and Microsoft to explore applications of holographic computing in space exploration. Earlier this year, NASA and Microsoft announced a collaboration to develop software called OnSight, a new technology that will enable scientists to work virtually on Mars using the same HoloLens technology.

NASA and Microsoft engineers tested Project Sidekick and the Microsoft HoloLens aboard NASA’s Weightless Wonder C9 jet to ensure they function as expected in free-fall in advance of their delivery to the microgravity environment of the space station. A pair of the devices is scheduled to launch on SpaceX’s seventh commercial resupply mission to the station on June 28.

“Microsoft HoloLens is about transforming the ways you create, connect, and explore. Sidekick is a prime example of an application for which we envisioned HoloLens being used — unlocking new potential for astronauts and giving us all a new perspective on what is possible with holographic computing” — Alex Kipman, technical fellow, Windows and Devices Group at Microsoft

Sidekick has two modes of operation. The first is “Remote Expert Mode,” which uses Skype, part of Microsoft, to allow a ground operator to see what a crew member sees, provide real-time guidance, and draw annotations into the crew member’s environment to coach him or her through a task. The second mode is “Procedure Mode,” which adds animated holographic illustrations displayed on top of the objects with which the crew is interacting.

Technological advancement is the bedrock of space exploration. Here we see augmented reality being used not for entertainment, but for true applications that could move humans into the next frontier.

Microsoft | NASA

Originally published at www.psfk.com on February 24, 2016.