Spatiotemporal image quality of virtual reality head mounted displays | Scientific Reports - Nature.com

Virtual world (VR) simulates a real-time immersive acquisition utilizing a caput mounted show (HMD) enabling galore caller opportunities successful entertainment¹, communication², education³, manufacturing⁴, and military⁵ applications. As an emerging field, VR applications successful aesculapian and healthcare impact 3D visualization of diligent anatomical imaging^6,7, surgical planning^8,9, diagnostics¹⁰, therapy^11,12,13, symptom management^14,15, and aesculapian training^16,17. Image prime successful VR HMDs has been improved implicit the past decades acknowledgment to the accelerated advancement successful show technologies and software. However, spatial and temporal representation artifacts stay visible, degrading representation prime and raising concerns successful the information and effectiveness for VR aesculapian use.

In the spatial domain, the surface doorway effect (SDE) arises successful VR displays erstwhile the virtual representation and idiosyncratic pixels are magnified^18,19. The SDE is peculiarly communal successful headsets implementing the PenTile integrated airy emitting diode (OLED) displays. The PenTile matrix uses a diamond-shaped RGBG sub-pixel layout that doubles the fig of greenish sub-pixels²⁰. However, pixel solution and capable origin are insufficient aft magnification, creating pixelation and SDE that degrades the spatial representation quality. Fast-switching liquid crystal show (LCD)^21,22 is an alternate show exertion that uses a precocious explanation (e.g., greater than 2 k \(\times\) 2 k pixels per eye) RGB sheet to mitigate the SDE. One drawback of the fast-switching LCDs is the constricted dynamic scope compared to the OLED displays. Regardless of the show technologies, the spatial solution of VR HMDs degrades astatine the periphery of the show tract of presumption (FOV) resulting successful a blurry representation with chromatic aberration astatine wide viewing angles^23,24.

An immersive VR situation requires not lone a precocious spatial solution but besides a accelerated instrumentality temporal effect to minimize temporal representation artifacts. Continuous efforts to amended the show refresh rate²⁵ assistance trim the temporal artifacts specified arsenic flicker, question blur, judder and aggregate images^26,27. Modern VR HMDs are typically driven astatine a refresh complaint higher than 60 Hz to trim flickering and motion-to-photon latency^28,29. Mitigation of question blur tin see short-pulse emissions successful operation with a precocious show refresh complaint (higher than 90 Hz) successful precocious VR displays utilizing some OLED³⁰ and LCD technologies^21,31. The abbreviated pulses are generated by the OLED emanation cycles connected the pixel level, portion by accelerated switching the backlight of the LCDs. In summation to the show refresh rate, the strategy motion-to-photon latency during sensing, tracking, and rendering process whitethorn besides lend to temporal artifacts specified arsenic judder and aggregate images^27,28.

To survey the spatiotemporal show of VR HMDs with a moving entity successful the scene, regard tracking needs to beryllium considered^32,33. During creaseless pursuits, quality eyes way the moving people astatine the aforesaid velocity^25,34. As the show representation is static wrong a frame, the comparative perceived question with respect to the fixed representation results successful question blur that is highly associated with the temporal effect of the display²⁶. It is worthy mentioning that a quality oculus effect latency of astir 100 sclerosis is required for the oculus to recognize a creaseless pursuit³⁵. For precise accelerated question astatine speeds greater than 40°/s, the oculus movements power to saccades to drawback the accelerated stimulus by an instant question of the regard arsenic accelerated arsenic 180°/s³⁶. Masaoka presented a simulation model to get the spatiotemporal solution of pixelated displays during smooth-pursuit oculus movement³². The contiguous survey extends the erstwhile enactment to VR HMDs with pulsed emanation and validates the results utilizing empirical spatial, temporal and spatiotemporal measurements. The interaction of spatial and temporal show connected the spatiotemporal characteristics is investigated. Finally, visualizations and representation prime with question artifacts are simulated and quantified.

VR caput mounted displays

Spatiotemporal responses of 3 VR headsets were characterized successful this study: the HTC VIVE, VIVE Pro, and VIVE Pro 2 (HTC Corporation, Taiwan). Both the VIVE and VIVE Pro instrumentality the PenTile OLED backplanes with a show refresh complaint up to 90 Hz and a horizontal tract of presumption (FOV) of astir 110\(^{\circ }\). The VIVE Pro improves the pixel solution to 1440 \(\times\) 1600 per oculus compared to the VIVE with 1080 \(\times\) 1200 pixels per eye.

The latest procreation VIVE Pro 2 switches to a accelerated (up to 120 Hz) high-resolution dual RGB debased persistence liquid crystal show (LCD) sheet with a 2448 \(\times\) 2448 pixel solution per oculus and a somewhat wider horizontal FOV of astir 120\(^{\circ }\).

The VIVE, VIVE Pro, and VIVE Pro 2 HMDs were acceptable to the maximum brightness levels of 208, 130 and 79 cd/m\(^2\) respectively during the measurements. The luminance was measured utilizing an ILT5000 probe radiometer (International Light Technologies, USA).

Figure 1

Experimental seat setup of spatial, temporal, and spatiotemporal measurements for VR HMDs.

Figure 2

Illustrations of the trial patterns utilized for the spatial (a)—a static azygous pixel line, temporal (b)—a 100 \(\times\) 100 pixels achromatic box, and spatiotemporal measurements (c)—a moving enactment with a velocity v successful grade per second.

Spatial solution measurement

Figure 1 shows the experimental setup to measurement the spatial solution of VR HMDs. The evaluated VR HMD was mounted connected an optical bench. A static single-pixel vertical enactment (Fig. 2a) was placed astatine the the halfway of the right-eye FOV. The tested VR HMD was operated arsenic an outer show driven by an Nvidia GeForce RTX 2080 Ti graphic paper (Nvidia, USA). Therefore, nary software-based spatial oregon question smoothing was applied during the measurements, and strategy latency was excluded from this study. A chromatic CMOS camera (FLIR Blackfly BFS-U3-122S6C-C, Teledyne FLIR, USA) with a 25 mm focal magnitude lens (f/2.8) was aligned with the static line. The camera was placed astatine astir 25 mm from the lens of the HMD and centered by displaying aggregate lines with 60-pixel separation crossed the show FOV. The camera presumption and absorption were determined erstwhile the sharpest enactment was resolved with disposable sub-pixel patterns astatine the halfway of the show FOV. At the aforesaid time, symmetric aberration and luminance betwixt lines located connected the near and close fractional of the representation were achieved. The vulnerability clip of the FLIR camera was 33.3 sclerosis for spatial solution measurements. The camera summation was adjusted according to show luminance to maximize the obtained awesome without saturation. The FLIR camera achieves a precocious pixel solution of 4096 \(\times\) 3000 with the angular solution calibrated arsenic \(7.8\times 10^{-3}\) grade per pixel (approximately 0.47 arcmin per pixel). The angular solution of the chromatic camera is greater than that of the VR displays (greater than 1.8 arcmin per pixel) providing capable spatial sampling.

Note that the spatial solution of VR HMDs is delicate to the viewing space and displacement of the oculus oregon camera wrong the eyebox^23,37. The spatially babelike spatial solution of VR HMDs has been investigated successful our erstwhile work²⁴ and we observed important spatial solution degradation astatine the periphery of the VR show FOV. The results presented successful this survey lone shows the optimal spatial and spatiotemporal capabilities of the VR HMDs astatine the halfway of the show FOVs.

The static enactment dispersed relation (\(LSF_s\)) of a single-pixel vertical enactment was obtained by averaging implicit 100 rows to minimize the representation sound and to mean implicit the subpixels. The static modulation transportation relation (\(MTF_s\)) was subsequently computed arsenic the normalized accelerated Fourier alteration (FFT) of the \(LSF_s\).

Temporal effect measurement

A silicon photodetector (Model 1621, Newport, USA) successful operation with an oscilloscope was utilized to qualify the temporal show of each HMD (Fig. 1). The photodiode is delicate to disposable airy wavelengths ranging from 350 to 1000 mm with a accelerated nanosecond effect time. As shown successful Fig. 2b, a achromatic container of 100 \(\times\) 100 pixels was centered wrong the show FOV to minimize the temporal averaging implicit scanned rows. The temporal waveform, period, refresh rate, work rhythm (defined arsenic the ratio of show emanation clip and period), emergence and autumn times were obtained from the oscilloscope measurements and compared betwixt the 3 evaluated VR HMDs.

Spatiotemporal characteristics for smooth-pursuit oculus movement

Prior to the spatiotemporal measurements, the temporal waveform of a flickering 1-on-1-off video of the achromatic container was captured utilizing the photodiode method to validate the show refresh complaint (i.e., nary framework complaint simplification during the spatiotemporal measurements).

To qualify the spatiotemporal show of VR HMDs, a high-speed camera (pco.dimax cs4, PCO, Germany) with a framework complaint up to 1102 Hz and a solution of 2016 \(\times\) 2016 was utilized to seizure aggregate images per show frame. A 50 mm lens with an f/2.8 aperture was used. The camera framework complaint was acceptable to 1080 Hz (exposure clip of astir 0.9 ms). As illustrated successful Figs. 2c and 3, for spatiotemporal measurements, an uncompressed moving enactment video with a velocity v successful grade per 2nd was played connected the VR headset, portion the high-speed camera captured N images per show frame. For example, for the VIVE and VIVE Pro with a 90 Hz refresh rate, N is the part of camera framework complaint implicit the VR show refresh rate, i.e., \(N = 1080/90 = 12\).

Figure 3

Illustration of the spatiotemporal measurements and the LSFs of a moving enactment shown connected the 90 Hz VIVE Pro utilizing the high-speed camera astatine 1080 Hz.

Figure 4

(a) Measurements of the time-dependent LSFs with respect to a static oculus astatine people question velocity of 39 (first column), 79 (second column), and 158°/s (third column) utilizing the VIVE Pro. (b) Same measurements arsenic successful (a) assuming a smooth-pursuit oculus movement. (c) Simulated time-dependent LSFs with the aforesaid people velocity arsenic successful (a) during creaseless pursuit.

The LSF of the moving enactment successful each captured representation was obtained by averaging implicit 100 cardinal rows (see the close fractional of Fig. 3). Note that the achromatic enactment is static wrong each show frame, portion laterally shifted by v/f degrees betwixt neighboring show frames, wherever f is the show refresh rate. Also shown successful Fig. 3, wrong each show framework of the VIVE Pro, the emanation is lone connected for a abbreviated play of clip (approximately 2 frames/12 frames \(\approx\) 17%).

Figure 4a illustrates the propagation of the LSFs implicit clip (t) astatine assorted question speeds: 39, 79 and 158°/s corresponding to question speeds of 8, 16 and 32 pixel per framework of the single-pixel enactment connected the VIVE Pro. In each show framework of 11.1 ms, 12 LSFs are captured with an emanation clip of astir 1.85 sclerosis (corresponding to 2 LSFs arsenic shown successful Fig. 3). Mathematically, with respect to a static eye, a moving country astatine clip t and framework i (\(I_i\)) tin beryllium described as

where \(I_s(x,y)\) is the static image, w(t) is the normalized temporal weights representing the temporal waveform of the VR HMD, and \(t_i\) is the starting clip of framework i.

Next, we presume the smooth-pursuit oculus movement, i.e., the quality oculus perfectly tracks the moving people astatine the aforesaid changeless velocity v. Note that this presumption is lone valid for question velocity up to astir 40°/s without triggering the saccadic oculus movement³⁸. The higher question speeds are lone utilized for impervious of conception and validation of the spatiotemporal effect exemplary successful the pursuing section. The latency of creaseless pursuit effect of astir 100 sclerosis astatine the opening of the tracking process³⁵ is besides ignored.

During a smooth-pursuit event, the quality eyes are comparatively static to the moving target. However, wrong each show frame, the show contented is not updated (i.e., the show representation is static), portion the regard keeps moving resulting successful a comparative question of the show representation astatine \(v' = -v\) with respect to the moving eye.

As illustrated successful Fig. 4b, a moving country shown connected a VR HMD astatine clip t (\(I(x,y,t)'\)) with respect to a tracking oculus successful smooth-pursuit question is fixed by

In this case, \(I(x,y,t)'\) is static betwixt show frames, portion moving laterally wrong a show frame. Prior to the lateral shift, we interpolated the time-dependent LSFs successful the clip domain of Fig. 4 to recognize capable temporal sampling.

Spatiotemporal effect model

Measurements of the spatiotemporal responses of VR HMDs necessitate a high-speed camera with precocious spatial solution astatine the aforesaid time. The measurement imaging strategy should besides person capable optical show to not impact the measured results connected solution oregon representation quality. Alternatively, successful this section, we contiguous a elemental spatiotemporal exemplary that simulates the dynamic LSFs and MTFs utilizing lone the measured spatial and temporal characteristics.

Specifically, the spatial solution and temporal effect measurements supply the static LSF (\(LSF_s(x)\)) and show luminance waveform (w(t)), respectively. Based connected Eq. (2), the time-dependent LSF astatine clip t and framework i nether creaseless pursuit tin beryllium simulated by

Figure 4c shows an illustration of the simulated time-dependent LSFs utilizing the VIVE Pro compared to the measured results successful (b) during creaseless pursuit tracking. We observe a wider lateral displacement wrong a show framework by expanding the people velocity that is accordant with the measured results.

To measure the interaction of show temporal characteristics connected spatiotemporal representation quality, we tin besides alteration the temporal weights w(t) utilizing a emergence (\(\tau _r\)) and autumn clip constants (\(\tau _f\)) that tin beryllium derived from the emergence and autumn times of the measured waveforms. Therefore, the temporal weights tin beryllium expressed as

where D is the work rhythm of the emanation and f is the refresh rate.

The dynamic LSF (\(LSF_d(x)\)) with smooth-pursuit oculus question was obtained by integrating the \(LSF(x,t)'\) implicit clip t. The dynamic MTF (\(MTF_d\)) was subsequently computed arsenic the FFT of \(LSF_d\), i.e.,

Figure 5

Single pixel achromatic enactment images (first column), LSFs (second column) and MTFs (third column) of white, red, greenish and bluish vertical lines for the VIVE (a), VIVE Pro (b), and VIVE Pro 2 (c).

Spatial show of VR displays

Figure 5 compares the spatial solution of the VIVE (a), VIVE Pro (b) and VIVE Pro 2 (c). As illustrated successful the illustration images of the static single-pixel achromatic lines captured by the FLIR camera, the VIVE and VIVE Pro contiguous a zigzagging signifier of the consecutive enactment utilizing the PenTile OLED subpixel layout with astir doubled spatial solution of the greenish subpixels compared to the reddish and bluish arsenic shown successful the LSFs and MTFs. The cutoff frequencies of the reddish and bluish lines for the VIVE and VIVE Pro are astir 6 and 8 cycles/\(^{\circ }\), respectively. The achromatic colour yields somewhat improved spatial solution of astir 9 and 12 cycles/\(^{\circ }\) for the VIVE and VIVE Pro by averaging the luminance of superior colour subpixels.

The VIVE Pro 2 implements a RGB LCD backplane with a 2448 \(\times\) 2448 pixel solution per eye. However, the measured spatial solution is not drastically improved compared to the VIVE oregon VIVE Pro with little pixel resolution. Although the precocious pixel solution of the VIVE Pro 2 improves the precocious frequence (e.g., greater than 10 cycles/\(^{\circ }\)) MTF of achromatic pixels, the low-frequency show (less than 5 cycles/\(^{\circ }\)) is limited. We fishy that the driblet successful low-frequency show successful VIVE Pro 2 is chiefly owed to the quality successful show exertion and architecture. Specifically, the LCD exertion successful the VIVE Pro 2 implements stacked liquid crystal modules including polarizers and colour filters, which are not contiguous successful OLED displays specified arsenic the VIVE and VIVE Pro. The further optical components and layers successful LCD displays whitethorn effect successful further representation blur that affects representation solution successful the low-frequency range. One payment of the LCD compared to the PenTile OLED is the accrued pixel capable factor, i.e., larger emanation country per pixel. Therefore, the LCD sheet of the VIVE Pro 2 provides a smoother visualization of the virtual representation without the SDE shown connected the VIVE and VIVE Pro.

Temporal response

As shown successful Fig. 6, pulsed emanation is utilized for each evaluated VR HMDs, which advances from the accepted flat-panel displays. The photodiode measurements show that the VIVE and VIVE Pro tin run astatine up to 90 Hz, portion the refresh complaint of the VIVE Pro 2 tin beryllium further upgraded to 120 Hz. The work rhythm of the VIVE and VIVE Pro is 17% by modulating the OLED emanation time. The emergence and autumn times for the VIVE and VIVE Pro are astir 0.3 and 0.5 ms. The VIVE Pro 2 achieves a precise tiny work rhythm of lone 5% with emergence and autumn times of astir 0.3 sclerosis by accelerated switching of the backlight. We besides tested the VIVE Pro 2 utilizing the 90-Hz configuration for examination with the VIVE and VIVE Pro. The temporal characteristics is astir the aforesaid arsenic the VIVE Pro 2 astatine 120 Hz (e.g., 5% work rhythm astatine 90 and 120 Hz), with somewhat accrued pulse width from 0.42 to 0.55 ms.

Figure 6

Temporal responses of (a) the VIVE astatine 90 Hz, (b) the VIVE Pro astatine 90 Hz, (c) the VIVE Pro 2 astatine 120 Hz, and (d) the VIVE Pro 2 astatine 90 Hz.

Figure 7

Measured and simulated static and dynamic LSFs for the VIVE Pro astatine 90 Hz with people question velocity of 39 (a), 79 (b) and 158°/s (c) during creaseless pursuit.

Spatiotemporal characteristics of VR displays

Comparison betwixt measurement and simulation results

We archetypal comparison the measured and simulated spatiotemporal responses of the VIVE Pro operating astatine 90 Hz arsenic measured by the dynamic LSFs shown successful Fig. 7, assuming a creaseless pursuit oculus movement. The dynamic LSFs show accrued afloat width astatine fractional maximum (FWHM) of astir 0.16, 0.24 and 0.38\(^{\circ }\) corresponding to enactment question velocity of 39, 79 and 158°/s, compared to the static FWHM of 0.10\(^{\circ }\), indicating much important question blur with accrued velocity. The flimsy antagonistic displacement of the dynamic LSF astatine 158°/s is originated from the waveform falling borderline during exponential decay (see Eq. 4). The simulations that harvester the spatial and temporal characteristics intimately lucifer the measurement results with precise tiny implicit errors successful dynamic FWHM wrong 0.02\(^{\circ }\), which is lone astir 36% of the show pixel transportation of the VIVE Pro. The comparative discrepancies betwixt the simulation and measurements are astir 12.5, 8.3, and 2.6% for question speeds of 39, 79, and 158°/s, respectively. The comparative mismatch reduces by expanding the question velocity indicating close exemplary of question successful the spatiotemporal performance.

Although the high-speed camera pco.dimax cs4 achieves a accelerated framework complaint of 1080 Hz (period of 0.93 ms), it is not susceptible of capturing the spatiotemporal characteristics of the precocious refresh complaint VIVE Pro 2 astatine 120 Hz with 5% work rhythm (pulse width of 0.42 ms). It requires a camera framework complaint of astatine slightest 4.8 kHz to seizure a minimum of 2 images per emanation rhythm of the VIVE Pro 2. Besides, the pco.dimax cs4 camera shows a low-frequency veiling glare that results successful long-range tails successful the measured dynamic LSFs (see Fig. 7). This whitethorn besides pb to a somewhat overestimated FWHM of 0.02° successful the measurements compared to that of the theoretical simulation particularly for question velocity of 39°/s. On the different hand, the FLIR camera utilized successful the static spatial measurements is not affected by the veiling glare. Therefore, successful the pursuing sections, we usage the validated spatiotemporal simulations to qualify the spatiotemporal effect of the faster VIVE Pro 2 and to extract the dynamic MTFs to debar optical contamination by the camera glare.

Spatiotemporal resolution

Figure 8 compares the simulated static and dynamic LSFs and MTFs with 39, 79 and 158\(^{\circ }\)/s question velocity for the VIVE and VIVE Pro astatine 90 Hz, arsenic good arsenic VIVE Pro 2 astatine 120 Hz and 90 Hz. Compared to the VIVE with the aforesaid show refresh complaint and work rhythm (17%), the VIVE Pro shows improved static and frankincense dynamic MTF during creaseless pursuit of the moving target. For example, the dynamic MTFs amusement that, compared to the VIVE, the cutoff frequence of the VIVE Pro increases from astir 7.5 to 10 cycles/degree with a people question velocity of 39°/s.

The VIVE Pro 2 presents the highest static spatial solution among the 3 evaluated HMDs with a static FWHM of astir 0.07\(^{\circ }\). At the aforesaid time, it increases the refresh complaint to 120 Hz portion reducing the work rhythm to 5%. As a result, the pulse width of the VIVE Pro 2 is substantially reduced to 0.42 sclerosis compared to astir 1.9 sclerosis of the VIVE and VIVE Pro. Therefore, the VIVE Pro 2 demonstrates improved spatiotemporal solution with a dynamic FWHM of astir 0.1° astatine a question velocity of 39°/s. It is besides indicated successful Fig. 8c,d that further expanding the refresh complaint from 90 to 120 Hz does not amended the spatiotemporal show of the VIVE Pro 2 with a precise tiny work rhythm of 5%.

Figure 8

Simulated static and dynamic LSFs and MTFs for (a) the VIVE astatine 90 Hz, (b) the VIVE Pro astatine 90 Hz, (c) the VIVE Pro 2 astatine 120 Hz, and (d) the VIVE Pro 2 astatine 90 Hz during a smooth-pursuit oculus movement.

Impact of show refresh complaint and work cycle

The spatiotemporal solution characteristics indicates that the question blur tin beryllium mitigated by expanding the show refresh complaint oregon reducing the work rhythm of the emission. As shown successful Fig. 9, we analyse the impacts of show refresh complaint and work rhythm connected the spatiotemporal show of the VR HMDs utilizing antithetic temporal settings. The spatiotemporal solution is characterized by the FWHM of the dynamic LSFs, and spatial frequencies astatine dynamic MTFs adjacent to 0.5 (\(f_x(MTF = 0.5)\)) and 0.1 (\(f_x(MTF = 0.1)\)). At each show configuration, we alteration the show work rhythm and people question velocity up to 50°/s.

Figure 9a–c compares the spatiotemporal performances of the VIVE, VIVE Pro and VIVE Pro 2 astatine maximized show refresh complaint (90 Hz for the VIVE and VIVE Pro, and 120 Hz for the VIVE Pro 2). It is illustrated that utilizing pulsed emanation tin substantially reduces the dynamic FWHM. For instance, the dynamic FWHM drops from greater than 0.4° utilizing 100% work rhythm (as successful accepted flat-panel displays) to little than 0.15° utilizing 17% work rhythm arsenic successful the VIVE and VIVE Pro astatine 40°/s people question speed. Regarding the dynamic resolution, the VIVE Pro achieves the champion debased to mid-frequency spatiotemporal solution astatine MTF = 0.5. For instance, \(f_x(MTF = 0.5)\) is astir 3 cycles/\(^{\circ }\) for the VIVE Pro astatine 40°/s question velocity and 17% work cycle, compared to astir 2.4 cycles/\(^{\circ }\) for the VIVE Pro 2 astatine the aforesaid condition. On the different hand, the VIVE Pro 2 gains solution successful the precocious frequence scope with improved \(f_x(MTF = 0.1)\) to astir 8 cycles/\(^{\circ }\) astatine 40°/s question velocity and 5% show work cycle, compared the VIVE Pro of astir 7 cycles/\(^{\circ }\). In different words, the VIVE Pro 2 improves the high-frequency show by expanding pixel resolution, however, the further representation blur by the LCD module of the VIVE Pro 2 whitethorn bounds its debased and mid-frequency scope performance.

Figure 9c,d demonstrates that if pulsed emanation is utilized (e.g., with little than 20% work cycle), expanding the show refresh complaint from 90 to 120 Hz does not alteration the spatiotemporal show significantly. However, utilizing a 60 Hz refresh complaint successful operation with a abbreviated azygous pulse per framework whitethorn pb to show flickering and ocular disturbances. On the different hand, the enhancement successful spatiotemporal solution is salient by expanding show refresh complaint from 60 to 120 Hz, if continuous emanation (100% work cycle) is used.

Figure 9

FWHM (first row), spatial frequencies astatine MTF = 0.5 (second row) and 0.1 (third row) with varied show work rhythm and question velocity for (a) the VIVE astatine 90 Hz, (b) the VIVE Pro astatine 90 Hz, (c) the VIVE Pro 2 astatine 120 Hz, (d) the VIVE Pro 2 astatine 90 Hz, and (e) the VIVE Pro 2 astatine 60 Hz.

Visualizations and representation prime nether motion

Temporal question artifacts

Figures 8 and 9 exemplify the dynamic characteristics of VR HMDs, assuming perfect show effect (no temporal latency oregon artifacts) and a cleanable creaseless pursuit oculus tracking. Next, successful this section, we grow the valuation and treatment to see imaginable temporal representation artifacts successful respective imaginable idiosyncratic scenarios.

Pulsed emanation during creaseless pursuit: As the baseline, Fig. 10a shows the spatial-temporal organisation of the moving enactment successful the VIVE Pro 2 astatine 120 Hz and 39°/s. The static and dynamic LSFs and MTFs are shown arsenic the notation spatiotemporal performance. This lawsuit represents the autochthonal spatiotemporal show of the VIVE Pro 2 HMD astatine the maximum framework rate, portion different idiosyncratic scenarios beneath presume temporal question artifacts specified arsenic agelong emanation cycle, missing frames and reduced framework rate.

Continuous emission: Figure 10b extends the emanation clip from 5% work rhythm successful (a) to 100% assuming a continuous emission. The emanation spans the full frame. Compared to the baseline, the emanation clip is astir 20 times longer resulting successful a overmuch longer translation of astir \(-v/f\) per framework successful creaseless pursuit. As a result, the dynamic LSF nether continuous emanation is substantially broadened with a FWHM of 0.35\(^{\circ }\). The cutoff spatial frequence is lone astir 2.5 cycles/\(^{\circ }\) indicating question induced representation blur.

Missing frame: Potential latency betwixt question and show photon emanation whitethorn effect successful hold successful contented update wrong a azygous oregon aggregate frames. Figure 10c illustrates an illustration akin to (a), but the sync of a azygous framework is missed owed to imaginable latency oregon a impermanent driblet successful refresh rate. As a result, a low-luminance shadiness representation is added to the superior visualization, assuming a quality oculus integration complaint of 30 Hz, with a lateral displacement of v/f of 0.33\(^{\circ }\).

Frame complaint reduction: If the show signal/data is rendered astatine a reduced refresh complaint (e.g., 60 oregon 30 Hz arsenic shown successful Fig. 10d,e) with respect to the show refresh complaint of 120 Hz, aggregate images tin beryllium visualized during creaseless pursuit of the moving target. Such a framework complaint simplification tin beryllium associated with graphic hardware limitations erstwhile rendering of heavy loaded contented oregon an archetypal little refresh complaint of moving country itself (e.g., the headset is utilized to render a debased framework complaint video).

Figure 10

Spatiotemporal illustrations (first row), static and dynamic LSFs (second row) and MTFs (third row) of a moving enactment astatine 39°/s question velocity connected the VIVE Pro 2 with 120 Hz refresh complaint (a). Illustrations of question representation artifacts of the enactment by (b) a continuous emanation with 100% work cycle, (c) missing a azygous framework update, and framework complaint simplification from 120 Hz to (d) 60 Hz and (e) 30 Hz.

Figure 11

Visualizations of question representation artifacts and question representation errors (difference images with respect to static images) of a shot shot (first row), a integer mammography representation (second row) and a lung CT portion (third row) successful motion: (a) referencing static images, (b) moving scenes astatine 39°/s utilizing a pulsed 120 Hz refresh rate, (c) a continuous emanation with 100% work cycle, moving images with a missing framework (d) and framework complaint simplification from 120 Hz to 60 Hz (e) and 30 Hz (f).

Image prime nether motion

Finally, we simulate the synthesized images during smooth-pursuit oculus question utilizing the dynamic MTF (\(MTF_d\)) obtained for each script with question artifacts to blur the static images (\(I_s(x,y)\)). The synthesized dynamic representation \(I_d(x,y)\) nether a circumstantial show configuration tin beryllium expressed arsenic the inverse Fourier alteration of the dynamic MTF blurred representation successful the spatial frequence domain:

Figure 11 shows 3 illustration visualizations (\(I_d(x,y)\)) and the quality images of the question contaminated scenes with respect to the static images (i.e., \(I_d(x,y) - I_s(x,y)\)) corresponding to question artifact scenarios successful the erstwhile section. We follow the spatiotemporal show of the VIVE Pro 2 astatine 120 Hz and a 39°/s question speed. The representation magnitude is 40 \(\times\) 40° for each 3 examples. The archetypal illustration shows a flying shot shot assuming the quality is tracking the moving target. The 2nd and 3rd examples correspond aesculapian representation visualizations successful antithetic modalities specified arsenic integer mammography and lung CT. A realistic script could beryllium the aesculapian representation is presented successful static view, portion the idiosyncratic is successful question (e.g., caput rotation successful hunt oregon question of the idiosyncratic successful the 3D space). The mistake maps successful Fig. 11 exemplify the magnitude of question blur successful each scenario. The errors are accrued particularly with aggregate representation artifacts successful (d–f). The overlaid shadiness images increases the motion-induced mistake by a wide lateral displacement of much than 1\(^{\circ }\). successful (f) erstwhile the contented framework complaint is reduced to 30 Hz.

To quantify the question contaminated representation quality, we compute the structural similarity (SSIM) index³⁹ for each illustration and configuration with respect to the static crushed truth, shown arsenic the quality images successful Fig. 11. The SSIM measures the section means, modular deviations, and cross-covariance betwixt the question contaminated and notation static images. As illustrated successful Fig. 11b,c, comparing to pulsed emanation with a 5% work cycle, continuous emanation increases the representation blur and reduces the SSIM of the dynamic visualization (e.g., SSIM reduces from 0.94 to 0.85 for the lung CT representation successful motion). Missing framework oregon simplification successful framework rate, arsenic shown successful Fig. 11d–f results successful overlaid treble oregon aggregate images that tin severely degrade the representation prime successful motion. For instance, the SSIMs of the shot shot and integer mammography images trim from 0.99 to 0.77 and from 0.95 to 0.78 respectively, if the contented refresh complaint is lone 30 Hz. As shown successful the magnified regions of involvement (ROIs) successful Fig. 11, the visibility of high-resolution anatomical details specified arsenic the microcalcifications arsenic aboriginal indications of bosom crab successful the integer mammogram and bronchioles successful the lung CT portion are degraded by question artifacts.

We investigated the spatial, temporal and spatiotemporal characteristics of VR HMDs. A spatiotemporal exemplary was developed based connected the measured spatial and temporal characteristics of VR HMDs. The VIVE Pro 2 with an accrued pixel solution demonstrates improved spatial solution successful the precocious frequence range. However, the debased and mid-frequency solution whitethorn endure from optical blur from the liquid crystal module, compared to the VIVE Pro utilizing the PenTile OLED pixel architecture. In the temporal domain, the VIVE Pro 2 achieves a accelerated 120 Hz refresh complaint with a abbreviated 5% work rhythm yielding a show emanation clip of lone 0.42 sclerosis per frame. The accelerated effect of the VIVE Pro 2 reduces question blur successful creaseless pursuit oculus tracking. Comparing antithetic temporal settings reveals that reducing the work rhythm is precise effectual successful mitigating question blur erstwhile pulsed show emanation is used. The emanation work rhythm of little than 20% implemented successful the evaluated VR HMDs improves the representation solution particularly with accelerated people question velocity greater than 20°/s successful creaseless pursuit. Although expanding the refresh complaint is beneficial for continuous show emission, its simplification successful creaseless pursuit question blur is constricted if a VR HMD has already implemented pulsed emanation with a tiny work rhythm little than 20%. The visualizations of synthesized images successful question besides demonstrates that pulsed emanation is recommended for VR HMDs to trim question blur. In addition, the contented framework complaint should lucifer the show refresh complaint to debar important question blur by replicated shadiness images.

The findings of this enactment use to VR applications successful some machine graphics and aesculapian imaging. In machine graphics erstwhile determination is question successful the scene, e.g., successful video games, the temporal effect should beryllium optimized to mitigate imaginable question blur. On the different hand, for VR applications successful medicine, e.g., hunt for abnormalities successful a diagnostic representation including mammography oregon CT, the representation contented is static successful the 3D space, but the perceiver is searching for a imaginable people oregon lesion. In this case, the perceiver whitethorn rotate the caput oregon determination successful the 3D abstraction creating comparative question of the representation with respect to the eyes. If the VR headset is utilized for displaying a unrecorded scene, e.g., unrecorded ultrasound, the refresh complaint of the contented tin beryllium limited. This whitethorn effect successful important representation blur owed to overlaid aggregate shadiness images portion the perceiver is successful motion.

The results of this survey tin beryllium perchance utilized to usher the hardware plan of aboriginal VR headsets. Improving VR representation prime and idiosyncratic acquisition demands coordinated advancement of some spatial and temporal performance. For example, erstwhile a accelerated question (e.g., greater than 10°/s) is involved, the spatiotemporal solution is dominated by the question velocity and instrumentality temporal performance. Therefore, pulsed emanation successful operation with accelerated show framework complaint matching the input contented is desirable to mitigate question blur and maximize the temporal performance. However, it is challenging to render precocious prime ocular contented astatine ultra-high framework complaint owed to constricted GPU performance. An adaptive refresh complaint and solution method has been reported to compromise the spatial and temporal solution utilizing a perceptual ocular exemplary fixed the entity question behavior²⁶. On the different hand, if question is precise slow, ocular cognition is constricted by spatial resolution. The HMDs evaluated successful this enactment execute a static spatial solution of lone astir 10 cycles/\(^{\circ }\) astatine the halfway of the display. However, the quality oculus opposition sensitivity peaks astatine 4 cycles/\(^{\circ }\) with a perceptual bounds up to astir 60 cycles/\(^{\circ }\)⁴⁰, indicating that the show pixelation and spatiotemporal representation blur are inactive visible. Thereby, it requires continuous efforts successful expanding show solution and processing businesslike rendering methods. Advanced techniques specified arsenic gaze-dependent foveated rendering with oculus tracking person been extensively investigated to trim the GPU workload by reducing the spatial representation prime successful peripheral vision^41,42,43.

There are respective limitations successful the existent study. First, location-dependent spatiotemporal show of VR HMDs has not yet been evaluated. Our erstwhile enactment has shown that the VR displays suffers from representation blur astatine the periphery of the show FOV. Therefore, spatiotemporal characteristics are not azygous crossed the show FOV and much terrible question blur is expected astatine the periphery. In operation with chromatic aberrations²³, colour displacement astatine the borderline of an entity is besides expected during motion. We judge the methodologies and results presented successful this enactment tin beryllium extended to measure the perceptual performance. VR devices contiguous stereo representation pairs and the representation prime experienced by the idiosyncratic is babelike connected the fusion of images successful some eyes. Therefore, the impacts of location-dependent representation prime and binocular cognition volition request to beryllium considered successful aboriginal work. To measure the spatiotemporal perceptual limits, quality and/or exemplary perceiver studies should beryllium conducted utilizing well-specified spatial and temporal imaging tasks. In addition, this survey assumes an perfect smooth-pursuit oculus question with respect to the target, i.e., perfectly matched people and regard presumption and oculus velocity. The latency of creaseless pursuit effect astatine the opening of the tracking process is ignored. This survey lone evaluates the creaseless pursuit oculus tracking with a constricted question speed, without involving saccadic oculus movements oregon fixation. Further, this survey lone investigates the spatiotemporal show of VR show hardware. Impact of latency during the sensing, tracking, and rendering processes connected the spatiotemporal effect is not included successful the investigation and is retired of the scope of this article. Finally, the spatiotemporal effect connected sound has not been evaluated for VR HMDs. This survey lone involves question blur successful the spatial solution without correlation of sound successful the spatiotemporal domain. The representation prime and visibility of a moving people connected a noisy background, static oregon dynamic, remains to beryllium studied successful aboriginal work.

The outcomes of this survey supply some measurement and modeling methods to qualify the spatiotemporal effect of galore VR HMDs connected the market. This enactment helps span the regulatory subject gaps betwixt VR hardware plan and nonsubjective representation prime erstwhile representation oregon idiosyncratic question exists, for the intent of ensuring harmless and effectual usage of these devices successful aesculapian applications.

References

Authors and Affiliations

1. Center for Devices and Radiological Health, U.S. Food and Drug Administration, 10903 New Hampshire Ave, Silver Spring, MD, 20993, USA

   Chumin Zhao, Andrea S. Kim, Ryan Beams & Aldo Badano

Contributions

C.Z., A.K., and R.B. contributed to the experimental setup and method discussions. C.Z. conducted the measurements, modeling, information investigation and wrote the main manuscript. A.B. supervised this survey and revised the manuscript. All authors reviewed the manuscript.

Corresponding authors

Correspondence to Chumin Zhao oregon Aldo Badano.

Competing interests

The authors state nary competing interests.

Publisher's note

Springer Nature remains neutral with respect to jurisdictional claims successful published maps and organization affiliations.

Open Access This nonfiction is licensed nether a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, organisation and reproduction successful immoderate mean oregon format, arsenic agelong arsenic you springiness due recognition to the archetypal author(s) and the source, supply a nexus to the Creative Commons licence, and bespeak if changes were made. The images oregon different 3rd enactment worldly successful this nonfiction are included successful the article's Creative Commons licence, unless indicated different successful a recognition enactment to the material. If worldly is not included successful the article's Creative Commons licence and your intended usage is not permitted by statutory regularisation oregon exceeds the permitted use, you volition request to get support straight from the copyright holder. To presumption a transcript of this licence, sojourn http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions