MPEG Immersive video (MIV)

The verification test report of MIV [1] was published.

[1] D. Mieloch (Ed.), Verification test report of MPEG immersive video, ISO/IEC JTC 1/SC 29/WG 04 N 0341, April 2023, Antalya.

At the 142^nd MPEG meeting, MPEG Video Coding (WG 04) promoted its MPEG immersive video (MIV) conformance and reference software standard (ISO/IEC 23090-23) to the Final draft international standard (FDIS) stage, the last formal milestone of its approval process. The document specifies how to conduct conformance tests and provides reference encoder and decoder software for ISO/IEC 23090-12 MPEG immersive video. This draft includes 23 verified and validated conformance bitstreams and encoding and decoding reference software based on version 15.1.1 of the Test model for MPEG immersive video (TMIV). The test model, objective metrics, and some other tools are publicly available at https://gitlab.com/mpeg-i-visual.

MIV was developed to support compression of immersive video content, in which a real or virtual 3D scene is captured by multiple real or virtual cameras. The standard enables the storage and distribution of immersive video content over existing and future networks, for playback with 6 degrees of freedom (6DoF) of view position and orientation. MIV is a flexible standard for multiview video with depth (MVD) and multi-planar video (MPI) that leverages the strong hardware support for commonly used video formats to code volumetric video. There are conformance bitstreams for all profiles, namely the MIV Main profile for MVD, the MIV Extended profile which enables MPI, and the MIV Geometry absent profile, which is suitable for use with cloud-based and decoder-side depth estimation. In addition to conformance testing, work on verification testing of MIV was finished. A formal subjective quality evaluation with “naïve” test subjects watching pre-defined pose traces in an immersive scene was performed. On average, MIV demonstrates a clear benefit over the previous state-of-the-art MPEG video standard for coding multiple views – the multiview extension of HEVC (MV-HEVC). According to the results, the average mean opinion score (MOS) savings of the MIV Main profile range from 1.20 to 4.69 (in 11-grade quality scale) for tested sequences, while for the MIV Geometry Absent profile the average savings was equal to 2.94.

Interdigital and Philips have collaborated to promote the MPEG-I standards ISO/IEC 23090-5 Video-based visual volumetric codec (V3C), ISO/IEC 23090-5 Annex H Video-based point cloud compression (V-PCC), ISO/IEC 23090-12 MPEG immersive video (MIV), and ISO/IEC 23090-14 Scene description at the AWE Europe conference [1] in Lisbon last week.

The promotion comprised:

• A booth on the exhibit with a set of related demos
• An oral presentation on the conference
• A press release about the collaboration
• Sponsorship of the conference

[1] https://www.awexr.com/

Leaflets: AWE_Leaflets.pdf

In July 2020, the Brazilian Terrestrial Television System Forum (SBTVD) issued a call for proposals to set the TV 3.0 standard [1], aiming at extending the current digital television system to new use cases and futuristic applications. Among these applications, it called for immersive (VR) codecs.

At the end of 2021, proposed MPEG V3C technologies (ISO/IEC 23090-5, Visual Volumetric Video-based Coding) were adopted to fulfill XR requirements for TV 3.0. The specification drafting is ongoing and complementary video real time tests (phase 3) are planned for H1/2023. The deployment of Brazil’s TV 3.0 system is planned for 2025.

Interdigital and Philips were present at the SET trade show and conference in Sao Paulo, Brazil (Aug 22nd to 25th 2022) to showcase technologies to support SBTVD standard 3.0 including immersive codecs. They share a booth with Fraunhofer (FhG).

The joint Interdigital/Philips demos were showcased, and panels were done for the promotion of the V3C standard towards many key players of Brazilian TV standards.

Flyer: Interdigital LinkedIn link, Philips link

The Motion Pictures Experts Group (MPEG), during its process of standardization, routinely conducts experiments to explore new technologies and to verify technologies that are introduced into a standard. The Video sub-group (WG04) also follows the same process. Experiments are performed on suitable video test material relevant to the experiment, where the test material are sourced from MPEG members and non-members alike. Without good test material, the outcomes of experiments could lead to inaccurate conclusions.

Currently, WG04, as part of its exploration activities, is performing experimentation on 6-DoF immersive video coding. In this activity, a three dimensional scene that is captured by numerous cameras (camera rig) is analysed and encoded. The arrangment of the cameras that capture the scene may have any arragement (regular/irregular). There is also no limitation placed on the inter-camera baseline distances; it may be narrow or wide. Each camera in the camera rig provides its captured videos in a known projection format, e.g. perspective or equirectangular.

While conducting exploration experiments on the above-listed topics, WG04 has realized that there is a dire need for more and appropriate test materials, especially omnidirectional (360 degree) 6DoF video content. There is also an immediate need for content that is captured/synthesized using non-planar rigs. Therefore, WG04 solicits new test materials from contributors. The call [1] describes the characteristics of test materials solicited, their formatting, and the process of contribution are described.

[1] Call for MPEG immersive video test materials, ISO/IEC JTC 1/SC 29/WG 04 N 0204, April 2022, Online.

The Motion Pictures Experts Group (MPEG), during its process of standardization, routinely conducts experiments to explore new technologies and to verify technologies that are introduced into a standard. The Video sub-group (WG04) also follows the same process. Experiments are performed on suitable video test material relevant to the experiment, where the test material are sourced from MPEG members and non-members alike. Without good test material, the outcomes of experiments could lead to inaccurate conclusions.

Currently, WG04, as part of its exploration activities, is performing experimentation on 6-DoF immersive video coding. In this activity, a three dimensional scene that is captured by numerous cameras (camera rig) is analysed and encoded. The arrangment of the cameras that capture the scene may have any arragement (regular/irregular). There is also no limitation placed on the inter-camera baseline distances; it may be narrow or wide. Each camera in the camera rig provides its captured videos in a known projection format, e.g. perspective or equirectangular.

While conducting exploration experiments on the above-listed topics, WG04 has realized that there is a dire need for more and appropriate test materials, especially omnidirectional (360 degree) 6DoF video content. There is also an immediate need for content that is captured/synthesized using non-planar rigs. Therefore, WG04 solicits new test materials from contributors The call [1] describes the characteristics of test materials solicited, their formatting, and the process of contribution are described.

[1] Call for MPEG Immersive video Test Materials, ISO/IEC JTC 1/SC 29/WG 04 N 0170, January 2022, Online.

At the 137th MPEG meeting, MPEG Video coding (WG 04) has promoted its MIV Conformance and Reference software standard (ISO/IEC 23090-23) to the Committee Draft (CD) stage. The document specifies how to conduct conformance tests, and it provides reference encoder and decoder software for ISO/IEC 23090-12 MPEG Immersive video. This draft includes 18 verified and validated conformance bitstreams, and reference software based on version 12.0 of the test model (TMIV). The test model, objective metrics and some other tools are publicly available at https://gitlab.com/mpeg-i-visual.

MIV was developed to support compression of immersive video content, in which a real or virtual 3D scene is captured by multiple real or virtual cameras. The standard enables storage and distribution of immersive video content over existing and future networks, for playback with 6 degrees of freedom of view position and orientation. MIV is a flexible standard for multiview video with depth (MVD) that leverages the strong hardware support for commonly used video codecs to code volumetric video. Views may use equirectangular, perspective or orthographic projection. By pruning and packing views, MIV is able to achieve bitrates around 15 to 30 Mb/s for HEVC and a pixel rate equivalent to HEVC Level 5.2. Besides the MIV Main profile for MVD, there is the MIV Geometry absent profile, suitable for cloud-based and decoder-side depth estimation, and the MIV Extended profile, which enables coding of multi-plane images (MPI). The MIV standard is designed as a set of extensions and profile restriction on the Visual Volumetric Video-based Coding and Video-based Point Cloud Coding (ISO/IEC 23090-5) standard. The conformance bitstreams span all profiles.

In addition to conformance testing, work on the verification test is ongoing, and the carriage of MIV is specified through the Carriage of V3C Data standard (ISO/IEC 23090-10). MPEG Requirements (WG 02) will publish final use cases and requirements for MIV edition-2, which will be an evolution of the MIV standard, and MPEG Liaison and Communication (AG 3) will publish the white paper on MIV.

Presenters:

Bart Kroon, Philips Research Eindhoven, Netherlands
Dawid Mieloch, Poznań University of Technology, Poland
Gauthier Lafruit, Université Libre de Bruxelles / Brussels University, Belgium

Abstract:

The tutorial gives a high-level overview of the MPEG Immersive Video (MIV) coding standard for compressing data from multiple cameras in view of supporting VR free navigation and light field applications. The MIV coding standard (https://mpeg-miv.org) is video codec agnostic, i.e. it consists of a pre- and post-processing shell around existing codecs, like AVC, EVC, HEVC and VVC. Consequently, no coding details like DCT block coding and/or motion vectors will be presented, but rather high-level concepts about how to prepare multiview+depth video sequences to be handled by MIV. Relations with other parts of the MPEG-I standard (“I” refers to “Immersive”), e.g. point cloud coding with V-PCC, and streaming with DASH, will also be covered.

Link to tutorial parts: Part 1, Demo break, Part 2 a, Part 2 b

In accordance to MPEG-I immersive standard roadmap, MPEG has submitted the Final Draft International Standard (FDIS) of MIV (ISO/IEC 23090-12) to ISO for publication. During the meeting, MPEG Video group has also progressed on the Verification Test definition which will allow the Advisory Group AG05 “MPEG Visual Quality Assessment” to assess the final status of the standard in early 2022 by verifying its performance compared to existing standards, namely in this case MV-HEVC. In parallel, the group has iterated on an improved working draft for MIV bitstream conformance in view of the ISO/IEC 23090-23 “Conformance and Reference Software for MPEG Immersive Video”. MPEG Video group has also updated use cases and requirements for a MIV 2nd edition and issued a new Call For MPEG immersive Video test materials.
Finally, MPEG Video group wants to advertise on its presence in a tutorial session dedicated to immersive video in general and MIV standard in particular at VCIP conference to be held in Munich from 5 to 8 December 2021. The 3 hours session will be presented by Bart Kroon from Philips Research Eindhoven, Dawid Mieloch from Poznan University of Technology and Gauthier Lafruit from Université Libre de Bruxelles.

At the 135^th MPEG meeting, MPEG Video Coding has promoted the MPEG Immersive Video (MIV) standard to the Final Draft International Standard (FDIS) stage. MIV was developed to support compression of immersive video content in which multiple real or virtual cameras capture a real or virtual 3D scene. The standard enables storage and distribution of immersive video content over existing and future networks for playback with 6 Degrees of Freedom (6DoF) of view position and orientation.

MIV is a flexible standard for multi-view video with depth (MVD) that leverages the strong hardware support for commonly used video codecs to code volumetric video. Views may use equirectangular, perspective or orthographic projection. By packing and pruning views, MIV is able to achieve bitrates around 25 Mb/s for HEVC and a pixel rate equivalent to HEVC Level 5.2. Besides the MIV Main profile for MVD, there is the MIV Geometry absent profile, suitable for cloud-based and decoder-side depth estimation, and the MIV Extended profile, which enables coding of multi-plane images (MPI).

The MIV standard is designed as a set of extensions and profile restriction on the second edition of the Visual Volumetric Video-based Coding (V3C) standard (ISO/IEC 23090-5). This standard is shared between MIV and the Video-based Point Cloud Coding (V-PCC) standard (ISO/IEC 23090-5 Annex H), and may potentially be used by other MPEG-I volumetric codecs under development.

The carriage of MIV is specified through the Carriage of V3C Data standard (ISO/IEC 23090-10).

Work on MIV conformance & reference software (ISO/IEC 23090-23) and verification tests is ongoing. The test model and objective metrics are publicly available at https://gitlab.com/mpeg-i-visual.

Finally, a number of so-called Exploratory Experiments have also been launched on diverse investigations such as refined depth generation, benchmarks with MV-HEVC (Multi-View High Efficiency Video Coding) or ML-VVC (Multi-Layer Versatile Video Coding), processing on specular materials or profile based on depth generation at the client side.