Author: Bertrand Chupeau

Presenters:

Bart Kroon, Philips Research Eindhoven, Netherlands
Dawid Mieloch, Poznań University of Technology, Poland
Gauthier Lafruit, Université Libre de Bruxelles / Brussels University, Belgium

Abstract:

The tutorial gives a high-level overview of the MPEG Immersive Video (MIV) coding standard for compressing data from multiple cameras in view of supporting VR free navigation and light field applications. The MIV coding standard (https://mpeg-miv.org) is video codec agnostic, i.e. it consists of a pre- and post-processing shell around existing codecs, like AVC, EVC, HEVC and VVC. Consequently, no coding details like DCT block coding and/or motion vectors will be presented, but rather high-level concepts about how to prepare multiview+depth video sequences to be handled by MIV. Relations with other parts of the MPEG-I standard (“I” refers to “Immersive”), e.g. point cloud coding with V-PCC, and streaming with DASH, will also be covered.

Link to tutorial parts: Part 1, Demo break, Part 2 a, Part 2 b

In accordance to MPEG-I immersive standard roadmap, MPEG has submitted the Final Draft International Standard (FDIS) of MIV (ISO/IEC 23090-12) to ISO for publication. During the meeting, MPEG Video group has also progressed on the Verification Test definition which will allow the Advisory Group AG05 “MPEG Visual Quality Assessment” to assess the final status of the standard in early 2022 by verifying its performance compared to existing standards, namely in this case MV-HEVC. In parallel, the group has iterated on an improved working draft for MIV bitstream conformance in view of the ISO/IEC 23090-23 “Conformance and Reference Software for MPEG Immersive Video”. MPEG Video group has also updated use cases and requirements for a MIV 2nd edition and issued a new Call For MPEG immersive Video test materials.
Finally, MPEG Video group wants to advertise on its presence in a tutorial session dedicated to immersive video in general and MIV standard in particular at VCIP conference to be held in Munich from 5 to 8 December 2021. The 3 hours session will be presented by Bart Kroon from Philips Research Eindhoven, Dawid Mieloch from Poznan University of Technology and Gauthier Lafruit from Université Libre de Bruxelles.

At the 135^th MPEG meeting, MPEG Video Coding has promoted the MPEG Immersive Video (MIV) standard to the Final Draft International Standard (FDIS) stage. MIV was developed to support compression of immersive video content in which multiple real or virtual cameras capture a real or virtual 3D scene. The standard enables storage and distribution of immersive video content over existing and future networks for playback with 6 Degrees of Freedom (6DoF) of view position and orientation.

MIV is a flexible standard for multi-view video with depth (MVD) that leverages the strong hardware support for commonly used video codecs to code volumetric video. Views may use equirectangular, perspective or orthographic projection. By packing and pruning views, MIV is able to achieve bitrates around 25 Mb/s for HEVC and a pixel rate equivalent to HEVC Level 5.2. Besides the MIV Main profile for MVD, there is the MIV Geometry absent profile, suitable for cloud-based and decoder-side depth estimation, and the MIV Extended profile, which enables coding of multi-plane images (MPI).

The MIV standard is designed as a set of extensions and profile restriction on the second edition of the Visual Volumetric Video-based Coding (V3C) standard (ISO/IEC 23090-5). This standard is shared between MIV and the Video-based Point Cloud Coding (V-PCC) standard (ISO/IEC 23090-5 Annex H), and may potentially be used by other MPEG-I volumetric codecs under development.

The carriage of MIV is specified through the Carriage of V3C Data standard (ISO/IEC 23090-10).

Work on MIV conformance & reference software (ISO/IEC 23090-23) and verification tests is ongoing. The test model and objective metrics are publicly available at https://gitlab.com/mpeg-i-visual.

Finally, a number of so-called Exploratory Experiments have also been launched on diverse investigations such as refined depth generation, benchmarks with MV-HEVC (Multi-View High Efficiency Video Coding) or ML-VVC (Multi-Layer Versatile Video Coding), processing on specular materials or profile based on depth generation at the client side.

In the 134th MPEG meeting, held between April 26th and April 30th of 2021, the MPEG Video Coding working group initiated work on verification tests and conformance. The group plans to publish a committee draft titled “ISO/IEC 23090-23 Conformance and Reference Software for MPEG Immersive Video” in a six month time-frame. Furthermore, in parallel with the finalising of the first edition of “ISO/IEC 23090-12 – MPEG Immersive Video”, a draft use-cases and requirements document for a second edition of MIV has also been published. In this draft, extensions to the first edition, such as handling of non-Lambertian surfaces and a larger viewing space, are proposed. An updated call for new visual test material will also be published.

As a result of the 133^rd MPEG meeting held January 11-15 2021, the ISO/IEC 23090-12 MPEG Immersive Video (MIV) specification has reached the Draft International Specification (DIS) stage, in conformance with a delivery to ISO expected end of this year.