The Vimmerse Freeport platform forms 3D video bitstreams from uploaded input video views. The input views are in an RGBD (RGB + Depth) format and are encoded as video in .mp4 or .mov files. Captures from any type of video cameras and/or depth sensors can be used, with any type of camera layouts, e.g. inward or outward facing cameras, using perspective or equirectangular projection. Projections rendered from graphics formats, such as mesh and point clouds, can be used as input views.

The platform creates two types of output bitstreams from the input file – bullet video and 3D video. Bullet video is a 2D video representation of the 3D video, following a pre-determined navigation path. 3D video gives viewers the ability to control navigation with 6 Degrees of Freedom (6DoF), where they can pan around or step into the scene. Bullet video may be streamed (HLS) or downloaded (MP4) for playback on any device. 3D video playback may be streamed (HLS) to the Vimmerse Freeport player.

Audio may optionally be used, with audio from a single view used in the output video bitstreams. Input audio can either be included with the captured video .mp4/.mov file or in a separate .mp3 or .aac audio file.

Content Format Description

One or more input views representing different perspectives of the same 3D scene can be processed to prepare the output bitstreams. For each input view, both color texture and depth must be available. Intrinsic camera parameters are required for each view which apply to both the color texture and depth for that view. If more than one view is uploaded, extrinsic camera parameters are also required for each of the views, which apply to both the color texture and depth for that view.

Simple content capture can be done with a single iPhone 12/13 Pro or ProMax, which contain a LiDAR sensor.

For content represented by multiple input views, a .json file is used to describe the views, including the camera intrinsic and extrinsic parameters and projection formats. When multiple input views are used, the views must be synchronized before upload.

Instructions for capturing and uploading content are available here.

Bullet video navigation path

The Freeport platform creates 2D bullet videos following a pre-determined navigation path with 6 degrees of freedom (6DoF). Creators may optionally upload a .CSV file describing the navigation path. If a navigation path CSV file is not uploaded, the platform will use a pre-set default pattern that exercises a range of movements are scaled within the covered viewing space of the capturing system when creating a bullet video. 

The CSV file contains a line for each frame in the video. Each line contains 6 values, representing the (X, Y, Z) position and (Yaw, Pitch Roll) orientation of the virtual camera for that frame.

An example navigation path, available here, is a crossed-square pattern where the navigation path starts from the center and spans the horizontal and vertical directions in cross motion then spans the edge boundaries of the capture system.

Current Platform Limits

For the current platform, the following constraints are imposed:

  • Max duration of processed video is 30 seconds or 900 frames.
  • Maximum composite video size is 4K x 2K. If the number of input views * resolution per view exceeds this limit, only a subset of the views are processed, to fit within the constraints.

Time limits may be imposed on how long prepared content is available for streaming or download.