Revolutionizing Object Detection for the Visually Impaired: Penn State's NaviSense Tool
The world of assistive technology for the visually impaired has seen remarkable advancements in recent years, but there's still room for innovation, according to researchers at Penn State. A groundbreaking tool, NaviSense, has emerged from their efforts, combining insights from the visually impaired community with cutting-edge artificial intelligence (AI).
NaviSense is a smartphone application that revolutionizes object detection for visually impaired users. It responds to spoken prompts, guiding users to objects in their environment through the phone's audio and vibrational feedback. During testing, NaviSense outperformed existing visual aids, earning the Best Audience Choice Poster Award at the Association for Computing Machinery's SIGACCESS ASSETS '25 conference in Denver. The tool's success was further solidified by its publication in the conference's proceedings.
One of the key challenges with existing visual aid programs is their reliance on in-person support teams, which can be inefficient and raise privacy concerns. Additionally, automated services often suffer from a critical flaw: the need to preload object models into the system's memory. This approach is inefficient and limits user flexibility. To overcome these limitations, the Penn State team integrated large-language models (LLMs) and vision-language models (VLMs) into NaviSense. These AI models enable the app to learn about its environment in real-time, recognizing objects based on voice commands without the need for preloading.
Ajay Narayanan Sridhar, a computer engineering doctoral student and lead investigator on the NaviSense project, emphasized the importance of user-centric design. The team conducted interviews with visually impaired individuals to understand their specific challenges and tailor the tool's features accordingly. This user-focused approach resulted in a conversational interface that filters objects based on verbal requests and asks follow-up questions for clarification, offering a level of convenience and flexibility that other tools struggle to match.
NaviSense also incorporates hand guidance, a feature that tracks the user's hand movements in real-time, providing feedback on the location of the desired object relative to the user's hand. This feature was a top request from survey participants, addressing a gap in existing solutions.
The effectiveness of NaviSense was demonstrated through a controlled environment test involving 12 participants. The tool significantly reduced the time users spent searching for objects and outperformed commercial alternatives in terms of accuracy and user experience. One user's positive feedback highlighted the tool's ability to provide precise location cues, making the search process more efficient and user-friendly.
While NaviSense is already effective and user-friendly, the Penn State team recognizes the potential for further improvements before commercialization. They are working on optimizing power usage to reduce the strain on the smartphone's battery and enhancing the efficiency of the LLMs and VLMs. The team's goal is to make NaviSense even more accessible and user-friendly, drawing on insights from testing and previous prototypes.
The development of NaviSense involved a dedicated team at Penn State, including Mehrdad Mahdavi, Penn State Hartz Family Associate Professor of Computer Science and Engineering, and Fuli Qiao, a computer science doctoral student. Additional contributions came from Nelson Daniel Troncoso Aldas, Laurent Itti, and Yanpei Shi, who provided valuable expertise in computer science and psychology. The project was supported by the U.S. National Science Foundation, underscoring the importance of continued research in this field.
As NaviSense continues to evolve, it holds the promise of significantly improving the lives of visually impaired individuals, offering a more efficient, accurate, and user-friendly approach to object detection and navigation.
