Multimorphing @ Theme Days "culture.digital.experience"

"Heritage 4.0" - Digitalization of Cultural Heritage

The Bavarian State Ministry of Science and the Arts aims to support employees and managers of museums, libraries, archives, theaters, and other cultural institutions in their digital transformation journey and has organized the themed event "kultur.digital.experience" from July 24 to July 26 for this purpose. We also participated in the innovation lab with our multimorphing technique to present our approach for generating and visualizing interactive, photorealistic digital twins of cultural institutions.

@ Leonardo Innovation Lab (© LEONARDO Zentrum, Felix Zeiss)

References

2023

Multidimensional Image Morphing - Fast Image-based Rendering of Open 3D and VR Environments

Simon Seibt, Bastian Kuth, Bartosz Von Rymon Lipinski, Thomas Chang, and Marc Erich Latoschik

Virtual Reality & Intelligent Hardware (Proceedings of CGI), 2023

Abstract Accepted and presented Available soon

The demand for interactive photorealistic 3D environments has increased in recent years and in various fields such as architecture, engineering and entertainment. Nevertheless, achieving a balance between quality and performance for high-performance 3D applications and Virtual Reality (VR) remains a challenge. This paper addresses this issue by revisiting and extending view interpolation for image-based rendering, enabling the exploration of spacious open environments in 3D and VR. Therefore, we introduce multi-morphing, a novel rendering method based on a spatial data structure of 2D image patches, called the image graph. With this approach, novel views can be rendered with up to six degrees of freedom using only a sparse set of views. The rendering process does not require 3D reconstruction of geometry, nor per-pixel depth information: All relevant data for output is extracted from local morphing cells of the image graph. Detection of parallax image regions during preprocessing reduces rendering artifacts by extrapolating image patches from adjacent cells in real-time. Additionally, a GPU-based solution to resolve exposure inconsistencies within a dataset is presented, enabling seamless transitions of brightness when moving between areas with varying light intensities. Experiments on multiple real-world and synthetic scenes demonstrate that the presented method achieves high “VR-compatible” frame rates, even on mid-range and legacy hardware, respectively.