Call for MPEG immersive video test materials

The Motion Pictures Experts Group (MPEG), during its process of standardization, routinely conducts experiments to explore new technologies and to verify technologies that are introduced into a standard. The Video sub-group (WG04) also follows the same process. Experiments are performed on suitable video test material relevant to the experiment, where the test material are sourced from MPEG members and non-members alike. Without good test material, the outcomes of experiments could lead to inaccurate conclusions.

Currently, WG04, as part of its exploration activities, is performing experimentation on 6-DoF immersive video coding. In this activity, a three dimensional scene that is captured by numerous cameras (camera rig) is analysed and encoded. The arrangment of the cameras that capture the scene may have any arragement (regular/irregular). There is also no limitation placed on the inter-camera baseline distances; it may be narrow or wide. Each camera in the camera rig provides its captured videos in a known projection format, e.g. perspective or equirectangular.

While conducting exploration experiments on the above-listed topics, WG04 has realized that there is a dire need for more and appropriate test materials, especially omnidirectional (360 degree) 6DoF video content. There is also an immediate need for content that is captured/synthesized using non-planar rigs. Therefore, WG04 solicits new test materials from contributors. The call [1] describes the characteristics of test materials solicited, their formatting, and the process of contribution are described.

[1] Call for MPEG immersive video test materialsISO/IEC JTC 1/SC 29/WG 04 N 0204, April 2022, Online.

Call for MIV test materials

The Motion Pictures Experts Group (MPEG), during its process of standardization, routinely conducts experiments to explore new technologies and to verify technologies that are introduced into a standard. The Video sub-group (WG04) also follows the same process. Experiments are performed on suitable video test material relevant to the experiment, where the test material are sourced from MPEG members and non-members alike. Without good test material, the outcomes of experiments could lead to inaccurate conclusions.

Currently, WG04, as part of its exploration activities, is performing experimentation on 6-DoF immersive video coding. In this activity, a three dimensional scene that is captured by numerous cameras (camera rig) is analysed and encoded. The arrangment of the cameras that capture the scene may have any arragement (regular/irregular). There is also no limitation placed on the inter-camera baseline distances; it may be narrow or wide. Each camera in the camera rig provides its captured videos in a known projection format, e.g. perspective or equirectangular.

While conducting exploration experiments on the above-listed topics, WG04 has realized that there is a dire need for more and appropriate test materials, especially omnidirectional (360 degree) 6DoF video content. There is also an immediate need for content that is captured/synthesized using non-planar rigs. Therefore, WG04 solicits new test materials from contributors The call [1] describes the characteristics of test materials solicited, their formatting, and the process of contribution are described.

[1] Call for MPEG Immersive video Test MaterialsISO/IEC JTC 1/SC 29/WG 04 N 0170, January 2022, Online.

Main outcomes of the 137th MPEG meeting (January 2022)

At the 137th MPEG meeting, MPEG Video coding (WG 04) has promoted its MIV Conformance and Reference software standard (ISO/IEC 23090-23) to the Committee Draft (CD) stage. The document specifies how to conduct conformance tests, and it provides reference encoder and decoder software for ISO/IEC 23090-12 MPEG Immersive video. This draft includes 18 verified and validated conformance bitstreams, and reference software based on version 12.0 of the test model (TMIV). The test model, objective metrics and some other tools are publicly available at https://gitlab.com/mpeg-i-visual.

MIV was developed to support compression of immersive video content, in which a real or virtual 3D scene is captured by multiple real or virtual cameras. The standard enables storage and distribution of immersive video content over existing and future networks, for playback with 6 degrees of freedom of view position and orientation. MIV is a flexible standard for multiview video with depth (MVD) that leverages the strong hardware support for commonly used video codecs to code volumetric video. Views may use equirectangular, perspective or orthographic projection. By pruning and packing views, MIV is able to achieve bitrates around 15 to 30 Mb/s for HEVC and a pixel rate equivalent to HEVC Level 5.2. Besides the MIV Main profile for MVD, there is the MIV Geometry absent profile, suitable for cloud-based and decoder-side depth estimation, and the MIV Extended profile, which enables coding of multi-plane images (MPI). The MIV standard is designed as a set of extensions and profile restriction on the Visual Volumetric Video-based Coding and Video-based Point Cloud Coding (ISO/IEC 23090-5) standard. The conformance bitstreams span all profiles.

In addition to conformance testing, work on the verification test is ongoing, and the carriage of MIV is specified through the Carriage of V3C Data standard (ISO/IEC 23090-10). MPEG Requirements (WG 02) will publish final use cases and requirements for MIV edition-2, which will be an evolution of the MIV standard, and MPEG Liaison and Communication (AG 3) will publish the white paper on MIV.

Tutorial VCIP 2021

Presenters:

Bart Kroon, Philips Research Eindhoven, Netherlands
Dawid Mieloch, Poznań University of Technology, Poland
Gauthier Lafruit, Université Libre de Bruxelles / Brussels University, Belgium

Abstract:

The tutorial gives a high-level overview of the MPEG Immersive Video (MIV) coding standard for compressing data from multiple cameras in view of supporting VR free navigation and light field applications. The MIV coding standard (https://mpeg-miv.org) is video codec agnostic, i.e. it consists of a pre- and post-processing shell around existing codecs, like AVC, EVC, HEVC and VVC. Consequently, no coding details like DCT block coding and/or motion vectors will be presented, but rather high-level concepts about how to prepare multiview+depth video sequences to be handled by MIV. Relations with other parts of the MPEG-I standard (“I” refers to “Immersive”), e.g. point cloud coding with V-PCC, and streaming with DASH, will also be covered.

Link to tutorial parts: Part 1, Demo break, Part 2 a, Part 2 b

Main outcomes of the 136th MPEG meeting (October 2021)

In accordance to MPEG-I immersive standard roadmap, MPEG has submitted the Final Draft International Standard (FDIS) of MIV (ISO/IEC 23090-12) to ISO for publication. During the meeting, MPEG Video group has also progressed on the Verification Test definition which will allow the Advisory Group AG05 “MPEG Visual Quality Assessment” to assess the final status of the standard in early 2022 by verifying its performance compared to existing standards, namely in this case MV-HEVC. In parallel, the group has iterated on an improved working draft for MIV bitstream conformance in view of the ISO/IEC 23090-23 “Conformance and Reference Software for MPEG Immersive Video”. MPEG Video group has also updated use cases and requirements for a MIV 2nd edition and issued a new Call For MPEG immersive Video test materials.
Finally, MPEG Video group wants to advertise on its presence in a tutorial session dedicated to immersive video in general and MIV standard in particular at VCIP conference to be held in Munich from 5 to 8 December 2021. The 3 hours session will be presented by Bart Kroon from Philips Research Eindhoven, Dawid Mieloch from Poznan University of Technology and Gauthier Lafruit from Université Libre de Bruxelles.

Main outcomes of the 135th MPEG meeting (July 2021)

At the 135th MPEG meeting, MPEG Video Coding has promoted the MPEG Immersive Video (MIV) standard to the Final Draft International Standard (FDIS) stage. MIV was developed to support compression of immersive video content in which multiple real or virtual cameras capture a real or virtual 3D scene. The standard enables storage and distribution of immersive video content over existing and future networks for playback with 6 Degrees of Freedom (6DoF) of view position and orientation.

MIV is a flexible standard for multi-view video with depth (MVD) that leverages the strong hardware support for commonly used video codecs to code volumetric video. Views may use equirectangular, perspective or orthographic projection. By packing and pruning views, MIV is able to achieve bitrates around 25 Mb/s for HEVC and a pixel rate equivalent to HEVC Level 5.2. Besides the MIV Main profile for MVD, there is the MIV Geometry absent profile, suitable for cloud-based and decoder-side depth estimation, and the MIV Extended profile, which enables coding of multi-plane images (MPI).

The MIV standard is designed as a set of extensions and profile restriction on the second edition of the Visual Volumetric Video-based Coding (V3C) standard (ISO/IEC 23090-5). This standard is shared between MIV and the Video-based Point Cloud Coding (V-PCC) standard (ISO/IEC 23090-5 Annex H), and may potentially be used by other MPEG-I volumetric codecs under development.

The carriage of MIV is specified through the Carriage of V3C Data standard (ISO/IEC 23090-10).

Work on MIV conformance & reference software (ISO/IEC 23090-23) and verification tests is ongoing. The test model and objective metrics are publicly available at https://gitlab.com/mpeg-i-visual.

Finally, a number of so-called Exploratory Experiments have also been launched on diverse investigations such as refined depth generation, benchmarks with MV-HEVC (Multi-View High Efficiency Video Coding) or ML-VVC (Multi-Layer Versatile Video Coding), processing on specular materials or profile based on depth generation at the client side.

Main outcomes of the 134th MPEG meeting (April 2021)

In the 134th MPEG meeting, held between April 26th and April 30th of 2021, the MPEG Video Coding working group initiated work on verification tests and conformance. The group plans to publish a committee draft titled “ISO/IEC 23090-23 Conformance and Reference Software for MPEG Immersive Video” in a six month time-frame. Furthermore, in parallel with the finalising of the first edition of “ISO/IEC 23090-12 – MPEG Immersive Video”, a draft use-cases and requirements document for a second edition of MIV has also been published. In this draft, extensions to the first edition, such as handling of non-Lambertian surfaces and a larger viewing space, are proposed. An updated call for new visual test material will also be published.