Be it a stylus which can turn onto an earpiece or an automated travel diary system, Microsoft has never ceased to amaze its audiences and customers. At the recently concluded Microsoft Inspire 2019, the tech giant showed a surreal technology known as Holoportation.
Researchers at Microsoft have been working on holoportation and mixed realities for a while now. Holoportation is a type of 3D capture technology that allows high-quality 3D models of persons to be reconstructed, compressed and transmitted anywhere, real-time.
Now, the researchers from Microsoft have been trying to make the approach of holoportation cellular by reducing the bandwidth, changing lighting and background, etc. and have successfully accomplished to a great extent.
Julia White, Azure Corporate Vice President demonstrated how holoportation works through Hololens 2 headset and the technologies used behind this innovation. She wore a Hololens 2 headset which generated a complete hologram of herself speaking in a foreign language.
Technologies Behind
The technology is developed using four approaches:
1| Mixed Reality: This technology is used to create the hologram of the speaker and render it in real-time at the event. Holoportation is a 3D capture technology which allows high-quality 3D models of people to be reconstructed, compressed and transmitted anywhere in the world in real-time. This technology is combined with mixed reality display such as, in this case, Hololens 2 headset in order to see, hear and communicate with people who are not present at that particular place.
2| Azure Speech-To-Text Capability: Azure Speech-To-Text enables real-time transcription of audio streams into text. This capability is used in the English transcription to extract the speech which can be consumed by the device and later take action on as command input.
3| Azure Translate: The Azure Speech Translator is a cloud-based automatic translation service which enables to create an end-to-end, real-time speech translation. This approach helps the device to translate the English speech into the Japanese language.
4| Neural Text-To-Speech Technology: Microsoft’s neural text-to-speech uses deep neural networks to overcome the limits of traditional text-to-speech systems in matching the patterns of stress and intonation in spoken language. In this case, the neural text-to-speech technology was applied to sound exactly the same way as White was. The only difference is that this time the 3D replica was speaking in the Japanese language.
The Demo
The demo was a jaw-dropping highlight from the event. The main idea behind this innovation is to fundamentally change the way of conversations with the persons who are miles away. It is basically a method of virtual teleportation from one space to another regardless of language and distance.
Azure Corporate VP, White said that the “game-changing” experience used mixed reality and Azure AI services to create the hologram of the speaker. The life-size hologram of White was created at Microsoft’s Mixed Reality Capture Studios. At the event, White put on the headset which created a smaller version of herself which she called “mini-me” and with the help of Hololens 2, she shows it off as if she is actually holding the smaller version of herself.
The speaker then commands “render keynote” and with the help of some special effects, the “mini-me” version is transformed into a real and full-sized hologram of the speaker. The hologram of White is rendered in real-time and spoke in the Japanese language with the speaker’s unique voice signature. The interesting part is that White personally does not know how to speak Japanese.
Watch the video here:
