Multi-dimensional image format

[Aaron Franke]

Hello AV1 folks,

I am interested in discussing the possibility of creating a new image format. Current image formats are focused on 2D images, they have a width and a height. However, there are many use cases where volumetric 3D content is required. This is common in many fields including computer graphics (ex: sampling in 3D or time-evolving 2D), data visualization, neuroscience (NIfTI), medical scans (DICOM for CT/MRI scans), and more. There are even use cases for 4D or higher textures (ex: time-evolving 3D).

Currently, most formats used to store 3D image data, such as neuroscience formats and KTX, operate by storing many layers of 2D images. This neglects optimization opportunities between multiple layers. 3D images require massive amounts of space compared to 2D images, therefore saving space is extremely important.

What I am searching for is an image format that can directly encode multi-dimensional images in an efficient manner, not layers of 2D images. Effectively, a single blob of data similar to AVIF, WebP, PNG, JPEG, etc.

I can see that part 10 of the JPEG 2000 standard allows for encoding of volumetric 3D image data, so maybe this does already exist. Here is a link to a research paper that explains this and explains the use cases for it: https://www.researchgate.net/publication/221376499_JPEG2000_Part_10_-_Volumetric_data_encoding

Would the AV1 folks, WebM project folks, or anyone else be interested in discussing or developing an optimized multi-dimensional image format? Even if JPEG 2000 part 10 can already encode 3D images, there is still the case of 4D images, 5D images, and so on. I'm sure that the AV1 team could come up with a better format if there was motivation to do this, like how AVIF is better than JPEG for 2D images.

Thanks,

Aaron Franke

[Aous Naman]

Hi Aaron,

For volumetric data, JPEG 2000 offers two coding solutions, these being
1) Multi-Component Transform (MCT) of JPEG 2000 Part 2.
2) JPEG 2000 Part 10, extensions for three-dimensional data.

The MCT option provided by JPEG 2000 Part 2, essentially considers each layer or slice of a volume as a "image component". Then by applying a Multi-Component Transform (MCT) it is possible to exploit the correlation that may be present across these layers or slices.
JPEG 2000 MCT can be applied at multiple levels, whereby the transformed slices can undergo additional transforms to further exploit redundancies between layers/slices within a neighbourhood. Furthermote, the MCT can be the discrete wavelet transform applied across layers/slices, which allows for additional features such as resolution scalability and random access.

For MCT, I am aware that Kakadu Software supports it, and possibly others.

JPEG 2000 Part 10 enables transforms across layers/slices to support 3D image coding. However we are unsure if an implementation exists.

Kind regards,

Aous.

PS: Feel free to contact me (here or at aous at unsw dot edu dot au) or Dr. Reji Mathew (reji dot mathew at unsw dot edu dot au ).