Investigator with NYU School of Engineering Heads Effort to Build Portable Pack so Blind Can Navigate Mass Transit System

For John Ross Rizzo, the task of improving navigation for those struggling with vision loss is personal.

The NY University professor recently joined the MTA Board in hopes of making public transportation more accessible for the visually impaired. Rizzo himself has firsthand experience with the struggle of commuting with weakened eyesight. “[He] has a rare medical disorder called choroideremia (CHM)... over the past 35 years, [CHM] has progressively narrowed his field of vision to a small straw-like tunnel in the very center” said the MTA in announcing he had been appointed to their board of directors.

His own work with NYU’s Tandon School of Engineering is making massive strides. Tandon recently landed a $5 million grant from the National Science Foundation towards a research project that aims to help the blind and those affected by low vision to use technology and AI to do things like navigating mass transit.

That in turn could afford many of the 285 million worldwide afflicted by blindness and low vision to join the workforce and fully participate in society. The NYU team said the grant will be used to advance progress on developing a prototype for a commercially available wearable device.

The NYC Department of Transportation and MTA are prominent government agencies involved with the drive to develop a commercially available project in the near future. Their corporate support is largely derived from Google, AT&T, and Tagged Web.

Navigating both urban and rural environments could ultimately make it easier for blind and low vision people to land jobs.

“People with blindness and low vision have unacceptably high unemployment rates, with some studies showing levels at about eighty percent,” said Rizzo, who is the project’s principal investigator. “A critical obstacle to employment is commuting difficulties and navigation within the workplace itself. This project takes a fundamental step in solving this problem. We believe, if successful, it can significantly improve quality of life and unlock potential for people with blindness and low vision.”

The technology has been named Visually Impaired Smart Service System for Spatial Intelligence and Onboard Navigation, or VIS4ION for short. The program works by using miniaturized sensors including cameras, microphones, GPS, and motion sensors on wearable devices to collect data about a user’s environment. Artificial Intelligence (AI) services, running locally within the platform itself and remotely in the cloud, process the sensor data to interpret the environment and tell the user where to walk, what to avoid and how to maneuver through hazards.

In a rarity, the grant will bring together several major schools at NYU that are already working on the project.

Rizzo is an associate professor in NYU Tandon’s Biomedical Engineering department, associate director of NYU WIRELESS, affiliated faculty at the NYU Tandon Center for Urban Science and Progress (CUSP) and associate professor in the Department of Rehabilitation Medicine at NYU Grossman School of Medicine.

”NYU has the rare ability to bring faculty from across schools and departments to collaborate on the creation of technologies that help create greater accessibility,” said NYU Tandon’s Dean Jelena Kovacevic in the university’s statement about the National Science Foundation grant. “Engineering can even the playing field and it is research like this, led by JR Rizzo, that ensure that everyone is able to navigate both urban and rural communities.”

The collaboration of researchers pulls talent from Departments of Mechanical and Aerospace Engineering, Civil and Urban Engineering, Biomedical Engineering, and Electrical and Computer Engineering. Research scientists and professors from NYU’s Grossman School of Medicine and Stern School of Business also join the engineers in diversifying the credentials behind the initiative.

The device collects information about the wearers surroundings by utilizing different motion sensors, microphones, GPS, and camera sensors. The information is then processed by an Artificial Intelligence service, which is able to translate the data from various sensors into instructions for the wearer on possible dangers to avoid and where to walk.

The National Science Foundation (NSF) had previously presented the VIS4ION team with a Phase One Convergence Accelerator grant in its earlier stages in 2022. After its initial success, the NSF was driven to continue aiding the project in a phase two funding round. These grants are extremely sought after and strive to confront problems surrounding disability.

The initial phase that was able to garner such a prestigious award was the creation of a lightweight VIS4ION backpack. Through the use of camera and audio processing, the prototype was able to effectively display the potential for the backpack to be used as a tool for safe navigation through various settings.

Furthering the success of their backpack prototype, researchers also developed VIS4ION mobile, a platform that can be used completely digitally on a smartphone. It can be used either just as a mobile platform or paired with the wearable device to construct 3D maps of detailed environments, both outdoors and indoors. Users are able to comprehend these maps through tactile and audio prompts, allowing them a better understanding of their surroundings.

The New York City Metropolitan Transit Authority (MTA) was also able to unite with VIS4ION to create a preliminary model of the Commute Booster. The product of the collaboration is an app which processes signage in subway stations in order to escort users through public transportation.

VIS4ION will be refined and improved throughout the future, with a focus on reducing the size and weight of the backpack. Meeting the goal of the team would mean a massive innovation that has not previously been accessible to users: a wearable product on the commercial market.

A significant partner in the project is Qualcomm, which continues to aid researchers in the research and development stages of the wearable device. Qualcomm is an affiliate of NYU WIRELESS, NYU Tandon’s 6g research center.

Dell Technologies joins Qualcomm as an NYU WIRELESS affiliate which aims to enhance the wearable through connections to cloud-based AI services.

The commercialization of the product is being directed by Tactile Navigation Tools (TNT), a start up partner. This partnership aims to increase the use of the technology to enhance the accessibility and quality of New York City’s real estate.

With the help of the National Science Foundation, researchers at NYU are able to emphasize the university’s dedication to their mission of solving challenges linked with health.