This project is targeting the retail industry, demonstrating how 5G enables new ways of interacting in shopping environments. Partners Sturfee and Mawari, with operators KDDI and Telefonica, envisioned using XR technologies to bring an unprecedented experience based on establishing co-presence: the feeling of being together in a place, despite being separated by great distance.

The user experience: a shopper in a physical store is joined by a shop assistant in a remote location. The assistant sees the shopper’s avatar inside the Digital Twin of the store, understanding the exact location and context, and can interact and communicate as if they are both present together in the retail store. The shopper, through their mobile phone’s Augmented Reality capabilities, sees the avatar of the shop assistant.

How it works

The partners establish this unique co-presence through a combination of innovative technologies.

Sturfee provides a Digital Twin platform for retailers to create, host, and connect digital twin shops to a physical store. The key element of the platform is the Visual Positioning Service (VPS) that connects the digital and physical spaces together, and synchronizes user activities across spaces.

The platform facilitates the creation of the digital twin shop from a 3D scan generated by Sturfee’s VPS powered mobile app. With VPS enabled inside the shop, AR app can determine the shopper’s position and orientation, rendering the shopper as a virtual character inside the digital space, while rendering the shop assistant in AR view with occlusion.

This project demonstrates shop assistants serving shoppers in a Ginza store, with Spanish-speaking shoppers assisted by the shop assistant (operator) in Spain, while the Japanese shopper assisted from Tokyo. The Mawari Network orchestrates the service in both countries, finding the right cloud rendering server for the XR-Tuber application. Mawari XR Streaming SDK allows to stream the XR-Tuber Application in Real-time to AR and VR simultaneously.

The Mawari XR-Tuber is a real-time system that captures facial motions through a smartphone and voice through a microphone, encodes it and then streams it to the Digital Human’s cloud rendering server where the lip-sync, rendering and streaming processes happen.

In other words, Mawari’s XR-Tuber provides the shop assistant with the tools to control and animate their avatar (Digital Human); Mawari’s XR Streaming SDK delivers the hyper-realistic 3D Digital Human avatar with the low latency & high-fidelity required to truly establish that feeling of being together.

Flow:

Step 1: The operator’s smartphone captures face data and sends it to PC with XR-Tuber installed. In parallel, the XR-Tuber application captures voice from a microphone.

Step 2: The XR-Tuber app synchronizes voice and face and streams this data together to the Digital Human application which is running in the edge cloud rendering server

Step 3: The digital human app processes XR-Tuber data and streams in real-time the result to AR and VR apps thanks to Mawari 3D Streaming Engine

Step 4: The End user device receives the stream and sees digital human speaking naturally with real expressions thanks Mawari XR Streaming SDK and to the operator’s voice and face gestures captured by the XR-Tuber Application

KDDI and Telefonica provide low latency connection and high bandwidth (5G connection) to the cloud rendering server that is streaming the Digital human to the end users.

Impact within the industry

This solution demonstrates how retailers can deliver better customer service by providing on-demand assistance by using remote shop assistance, and co-presence. Rather than waiting for assistance from an assistant actually in the physical location, shoppers are able to access help from the remote assistant instantly. Through the contextual awareness of co-presence, that assistant feels as if they are with the shopper; they each can know what the other is looking at, and what’s around them in the store.

We have built these capabilities on the unique improvements that 5G brings in terms of lower-latency, higher bandwidth and edge compute nodes, creating the innovative ability to establish presence together in a real world location.

This basic co-presence will have so many applications for social good, by bringing together people into a real world location as never before possible. In our first instance here, we address urgent disruptions to the retail labor markets. Covid’s impacts have made it difficult in many markets to find local help; with our combined solution we are able to connect workers farther away with job opportunities.

The principle can work in retail equally well with roles reversed. Shoppers can stay at home, visiting shops anywhere in the world, with an opportunity to truly interact and transact as if there. People can explore locations and cultures from afar, and then plan visits to those areas and peoples that they find most appealing. Most importantly, this can bridge the gap between online and offline commerce for many small businesses, who have so far been unable to compete with large e-retailers. By taking advantage of their real world location, and making that location more accessible for people at a distance, these small local businesses can thrive.

Finally, these capabilities will enable our societies to fulfill our environmental mandate to travel less frequently, expending less carbon, yet still get all the benefits of being together in face to face situations.

Using near-real time 5G networks, Mawari’s technology enhances the co-presence experience through delivery of a high-fidelity avatar that has the actual facial expression of a real person. Combined with voice communication, the interaction between shopper and assistant becomes much more natural, expressive and useful.

Expected Results

1. Ability to orchestrate a shared experience / presence between two continents and two different operators in the real and virtual world using 5G, edge computing / Cloud rendering and XR Streaming technologies. Note: This project trials the retail use case, which is an example, and the technologies can be used for any other use cases providing a shared experience / presence in the real and virtual world.

2. Determine by testing different network profiles, what is the minimum requirement for this type of service to run on LTE and 5G networks, in order to implement Quality on Demand (QoD) network parameters/API in the near future.

Project Resources