Fudan University Develops "Hearing the World" App to Empower Visually Impaired with Multisensory Perception of the World

2024-03-04

Fudan University's Natural Language Processing Laboratory (FudanNLP) has recently launched an innovative app called "Hearing the World" after continuous efforts by its faculty and students. This app, based on the multimodal large-scale model "Fudan・MouSi," introduces a new way for visually impaired individuals to perceive the world. The "Hearing the World" app only requires a camera and a pair of headphones to transform visuals into vivid language descriptions, helping visually impaired individuals better understand and perceive their surroundings. In addition to depicting scenes, the app also provides real-time alerts for potential risks, ensuring the safety of visually impaired individuals. To meet the diverse needs of visually impaired individuals in their daily lives, the "Hearing the World" app offers three practical modes: 1. Street Walking Mode: In this mode, the app meticulously scans the road conditions to provide accurate traffic information and potential risk alerts, ensuring safe navigation. 2. Free Questioning Mode: This mode allows visually impaired individuals to easily explore places such as museums, art galleries, and parks, capturing every detail of their surroundings. By constructing a rich soundscape, the app enables visually impaired individuals to enjoy a colorful world. The official demonstration also shows advanced features, such as transcribing TV screen content. 3. Object Finding Mode: This mode provides visually impaired individuals with the ability to quickly locate everyday objects, serving as a reliable assistant and making their lives more convenient. It is reported that the "Hearing the World" app is expected to complete its first round of testing in March this year and will be piloted simultaneously in China's first and second-tier cities and regions. Depending on the deployment of computing power, the app will gradually expand to more areas, bringing blessings to more visually impaired individuals. This innovative achievement not only demonstrates Fudan University's leading strength in the field of natural language processing but also contributes warmth and care to society.