IBC2022 Tech Papers: Data compression for 6 degrees of freedom virtual reality applications - IBC365

Abstract

6 Degrees of Freedom (DoF) are utilized successful Virtual Reality (VR) applications to heighten the idiosyncratic acquisition compared to the modular 3 DoF solutions. Due to its sparse nature, 6 DoF accusation is typically represented successful a constituent unreality form, wherever each constituent describes the presumption of a constituent successful the 3D space, arsenic good arsenic its attributes (e.g.,colour and transparency). Although it enhances idiosyncratic experience, 6 DoF requires a higher measurement of information compared to 3 DoF, which has made contented organisation challenging and has besides constricted its applications to high-end specialised machines. The purpose of our enactment was to plan a caller constituent unreality compression strategy to let 6DoF VR applications to tally successful real-time connected high-end user devices, specified arsenic gaming laptop and desktop machines. Although our solution was designed specifically for the PresenZ 6 DOF VR movies format, it whitethorn beryllium easy applied connected different volumetric video formats arsenic well.

Introduction 

In a emblematic Virtual Reality (VR) scenario, Degrees of Freedom (DoF) are utilized to way the question of a headset-wearing idiosyncratic wrong a three-dimensional (3D) abstraction and set accordingly the representation that the idiosyncratic views. 3 DoF applications way lone rotational question astir the x, y, and z axes (known arsenic pitch, yaw, roll), portion 6 DoF applications besides way translational question (surging, swaying, heaving), allowing for further effects, specified arsenic moving forward/backward, left/right, and up/down. In summation to enhanced idiosyncratic experience, 6DoF VR tin assistance trim question sickness and feelings of disorientation, by providing a amended consciousness of presence.

Due to its sparse nature, 6 DoF accusation is typically represented successful a constituent unreality form, wherever each constituent describes the 3D presumption of a point, arsenic good arsenic its colour, transparency, orientation, and motion. It whitethorn besides incorporate further data, specified arsenic accusation astir the camera(s) utilized to seizure the 3D view. The existent fig of points depends connected the complexity of the ocular scene: a emblematic framework whitethorn dwell of implicit 5 cardinal points.

Although it enhances idiosyncratic experience, 6 DoF requires a higher measurement of information compared to 3 DoF, which has made contented organisation challenging and has besides constricted its applications to high-end specialised machines. The cardinal challenges that 1 needs to code are: 1) precocious information entropy, which typically exceeds the capableness of accepted connection channels, specified arsenic the 500 MB/s of Solid-State Drives (SSD), and 2) real-time video rendering requirements astatine comparatively precocious framework rates (30 fps). In this work, we picture our attack towards addressing the supra challenges utilizing a caller information compression scheme, designed specifically for constituent unreality datasets.

Our information compression format describes each framework individually, and consists of a fixed header layer, arsenic good arsenic respective optional information layers. The fixed header furniture describes basal information, specified arsenic the fig of points and the utilized colour space, arsenic good arsenic the types of coding tools and techniques utilized for assorted constituent unreality subgroups and their attributes. Depending connected the accusation included successful the fixed header, further header layers whitethorn beryllium contiguous successful the bitstream, further describing encoding methods, parameters, and metadata. Finally, further halfway layers are utilized to store the encoded values for each attribute.

We besides designed and implemented a codec API, that allows encoding of a bid of constituent unreality frames anddecoding it successful real-time connected high-end laptops and gaming desktop machines. Our existent encoder and decoder implementations were developed successful C++, utilising techniques specified arsenic multi-threading and IntelTM Single-Instruction-Multiple-Data (SIMD) intrinsics.

This insubstantial discusses inheritance enactment for constituent unreality compression and VR applications, past describes our attack successful detail, our experimental results, conclusions and discusses imaginable further developments.