NYU Tandon cuts a rug with new 3D video technology

A groundbreaking research project spearheaded by NYU Tandon School of Engineering will bring cutting-edge immersive three-dimensional (3D) video to dance education, making learning the art form more available to a diverse range of students.

The project, supported by a new $1.2 million four-year grant from the National Science Foundation, aims to make Point-Cloud Video (PCV) technology suitable for streaming onto standard Internet-connected devices.  

Developed over the past decade, PCV is a type of highly-detailed 3D video that allows viewers to “move” within an immersive video environment, observing objects and scenes from any angle or distance as if they were physically present. Unlike traditional video that relies on two-dimensional pixels,  PCV captures scenes as point clouds, collections of data points in 3D space.

Currently, streaming PCVs requires bandwidth and computational power that exceeds what is generally available, limiting its real-world applications. The NYU Tandon team’s project will address those obstacles, reducing bandwidth consumption and delivery latency, and increasing power consumption efficiency so that PCVs can be streamed far more easily. 

“With recent advances in the key enabling technologies, we are now at the verge of completing the puzzle of teleporting holograms of real-world humans, creatures and objects through the global Internet,” said Yong Liu, professor of electrical and computer engineering (ECE), who leads the research team that includes ECE Professor Yao Wang and R. Luke DuBois, associate professor in the Technology, Culture and Society Department. 

“The success of the proposed research will contribute towards wide deployment of high quality and robust PCV streaming systems that facilitate immersive augmented, virtual and mixed reality experience and create new opportunities in many domains, including education, business, healthcare and entertainment,” said Liu.

As part of the project, DuBois is heading up a collaboration with the Mark Morris Dance Group’s Dance Center – a cultural hub in Brooklyn that serves 1,500 students – and the Department of Dance at NYU Tisch School of the Arts, to test PCV streaming with human motion. 

“Point-Cloud Video holds tremendous potential to transform a range of industries, and I’m excited that the research team at NYU Tandon prioritized dance education to reap those benefits early,” said Jelena Kovačević, NYU Tandon Dean.  “NYU encourages cross-disciplinary research, and that means bridging engineering with new fields. Collaborating with dancers from NYU Tisch and Mark Morris makes the innovative power of engineering imminently real.

Dancers from both organizations will perform on a volumetric capture stage — a specialized production environment for shooting 3D video — at NYU Tandon @ the Yard, the school’s production and research facility for emerging media at the Brooklyn Navy Yard. Their movements will be streamed live and on-demand, using the new PCV testbed the researchers develop. 

“By facilitating the use of emerging technologies, NYU Tandon is laying the groundwork for greater accessibility in the dance and education spaces while broadening the creative possibilities for artists,” said Allyson Green, NYU Tisch School of the Arts Dean. “As a longtime choreographer, performer, and dance educator, I am thrilled that our Department of Dance will be joining NYU Tandon and Mark Morris Dance Group at the forefront of this research, and I’m eager to see how these advancements modernize the field.”

Students watching the 3D streams will be able to observe the dancers from varying angles and distances, necessary for learning dance techniques and choreography properly. 

“Integrating dance education into our PCV research brings to life the real-world benefits of this technology, demonstrating how it can open doors for people everywhere to learn things that previously would have been possible only in person,” said DuBois. “Our project also exposes dancers to an understanding of the possibilities and challenges of 3D video, giving diverse students and practitioners insight into the ways in which computers, engineering and media positively impact their work and arouse their interest in computing.”

“We are excited to embark on a dynamic collaboration with the exceptional team at NYU Tandon, and look forward to exploring the potential and innovation this technology promises to bring to the realms of dance education and creative expression,” said Sarah Marcus, Director of Education and Community Engagement, Mark Morris Dance Group.

From a technical perspective, beyond developing the PCV testbed used specifically for dance experimentation, the NYU Tandon researchers want the project to make fundamental strides in three areas that will advance PCV more broadly.

First, they will develop efficient PCV compression using object-based coding schemes, hierarchical slicing structures, and advanced prediction techniques that enables bitstream adaptation based on the predicted viewer’s field of view and the network throughput. Second, they will create a progressive streaming framework that refines spatial resolution based on viewer needs, considering computation-communication trade-offs and low-latency live streaming. Lastly, they will design edge-based caching algorithms, multi-user delivery strategies, and cross-user Field-of-View (FoV) predictions to enhance PCV streaming efficiency and robustness.

Mark Morris Dance Group and the Department of Dance at NYU Tisch will recruit students and instructors to take part in the project, and will start performing on the volumetric capture stage in Spring 2024. The project officially started on October 1, 2023. 

This project builds on NYU Tandon’s advanced work to build the next generation of foundational wireless communications and media processing technologies, as well as extensive industry collaborations with leaders in immersive video and virtual production, including Evercoast, Final Pixel, Argus, Epic Games, and Walt Disney, to bring research-grade emerging technology within reach of media, entertainment and cultural institutions while developing technologists who are exceptionally prepared to further the field of emerging media.