iPhone Capture using Vimmerse Capture app

Contact us at info@vimmerse.net to get access to the Vimmerse Capture app for iPhone 12/13 Pro or ProMax models on TestFlight.

  1. Install and launch the app.
  2. Login to your Vimmerse account on the app. Create an account if needed on a browser at https://vimmerse.net, and selecting Login in the upper right.
  3. Mount iPhone on a tripod or otherwise put in a stationary position.
  4. Select the Record icon to begin recording, and the stop button to stop recording.
  5. Optionally use the RGB and Depth buttons to switch between viewing the color and the depth.
  6. Select the app Gallery button to view captured content within the app or access via Photos
  7. Select the app Upload button to upload content directly to the Vimmerse platform. Alternatively, upload the file (located in Photos) from the browser at https://vimmese.net, Login to your account, and select “Upload new content” button on the right. Drag and drop all capture files (from step 1 above), select the capture content format, and select Upload. Wait for uploading of files to complete and select “Submit for Processing”.

iPhone Capture using Record3D app

The iPhone 12/13 Pro or ProMax models are supported, using a 3rd party capture app, Record3D. 

  1. Download the Record3D app from the App Store.  https://apps.apple.com/us/app/record3d-3d-videos/id1477716895 . Purchase the Basic app and the RGBD extension.
Graphical user interface, text, application, chat or text message

Description automatically generated
  1. Mount phone on a tripod or otherwise put in a stationary position
  2. In the Record3D Settings tab, set the Animation Rate(FPS) to 30
  3. Recommended: In the Record3D Settings tab, select Export options, turn on “Autoexport mp4 videos to the Camera Roll”
Graphical user interface, text

Description automatically generatedGraphical user interface, text

Description automatically generated
  1. Use Record3D to record a video clip by using the red record start/stop button on the Record tab, or consider using an Bluetooth remote control (such as https://www.amazon.com/Shutter-Control-Bluetooth-Wireless-Technology/dp/B07YSNZLC6 ) to start/stop recording. It is recommended to begin or end the capture of the background scene without any people in it, for improved 3D video quality.
  2. In the Library tab, select Edit, select the video clip, select Export, and select Looking Glass (.mp4)
A screenshot of a car

Description automatically generated with medium confidenceGraphical user interface

Description automatically generated with low confidenceText

Description automatically generated

If you followed recommended step 4, the exported file will be in Photos, otherwise located the file by browsing to Files, Record3D, and then to a Folder whose name is the capture date & time, then to the LookingGlass folder. It is also possible to edit the .mp4 with the built-in iOS video editor to adjust the start/stop of the video. (The iOS video editor output is mov format, which can also be uploaded.)  

Login to the Vimmerse platform in a browser, select Upload content. In the Capture Device pulldown, select iPhone.  Upload the .mp4 (or .mov) video you captured.  Select the rotation angle (0°, 90°, 180° or 270°) in the pulldown to indicate the portrait/landscape mode the video was captured in, as illustrated in the figure below.

Azure Kinect DK Capture using DepthKit

Capture using an Azure Kinect DK is supported with use of DepthKit. https://www.depthkit.tv/

1. Record: Use DepthKit to record a scene, following the instructions in the DepthKit documentation at https://docs.depthkit.tv/docs

2. Export :

  • Select the EDIT tab
  • Select REFINE
    • Select Enable refinement
    • Set the Filter Size and Fill Amount to the maximum values
  • Select ISOLATE
    • Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene. Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
  • Select EXPORT
    • Select Combined Per Pixel video
    • Export

3. Upload: Login to the Vimmerse platform in a browser, select Upload content. In the Capture Device pulldown, select Azure Kinect – DepthKit. Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project. On the Upload content page, upload the .mp4 and .txt files for your capture.

Multi-view inputs

For multi-view inputs, multiple files must be uploaded.  An example set of files is available for download here.

A single .json file is used to provide parameters describing the each of the views, including the camera parameters and depth range. 

A separate .mp4 or .mov file (8 or 10 bit yuv420 format) is used for color texture or depth of a single view. A naming convention must be followed for the .mp4/.mov files to identify the view number and whether color texture or depth is represented.  

Normalized disparity is used to represent metric depth, which allows finer representation for close objects than for distant objects or background. Metric depth, z, is mapped to quantized normalized disparity, d, as:

d = (1/z – 1/zfar)/(1/znear – 1/zfar) * (2bitdepth -1 )

where [znear, zfar] represents the depth range in scene units and bitdepth is the number of bits per sample. The depth map represents the axial depth (i.e. normal distance of points with respect to image plane) for perspective projections and represents the radial depth (i.e. absolute distance of points to the camera’s center) for equirectangular projections.      

If inputs are represented with a series of still images in png or jpeg format, ffmpeg can be used to combine the series of images into a single mp4 video, encoded as either H.264/AVC or H.265/HEVC. The example command below can be used as a guide.

ffmpeg -r 30 -s:v 1920×1080 -i pic%04d.png -vcodec libx265 -crf 10 -pix_fmt yuv420p test.mp4

The following naming convention is to be used for the files:

  • base.json: sequence parameters (Any name can be used in place of “base”)
  • v0_t.mp4: texture video for view 0
  • v0_d.mp4: depth video (as normalized disparity) for view 0
  • base.aac or base.mp3 : audio  (optional)

Example test content suitable for uploading can be found here. The example is for 4 views of the Frog sequence @30fps and scaled down to 640×480. Frog is an MPEG MIV test sequence and provided by Intel Corporation, available under the Creative Commons 4.0 License.

A detailed description of the sequence parameters .json file can be found here.

Android Capture

To be supported in upcoming release