People with Visual Impairments (PVI) often have difficulty interpreting their surroundings correctly due to the lack of visual cues. Understanding a scene through tactile exploration proves insufficient and potentially hazardous. Often people need to recognize the whole scene at first glance, then gaze at each object, sort out their relationships, and react to the scene.

We explore scene understanding for helping PVI. In this work, we design an Open Scene Understanding (OpenSU) system which consists of Grounded Situation Recognition (GSR) and Segment Anything (SAM), allowing PVI to perceive the entirety of the scene and retrieve object information in a specified direction.

Grounded situation recognition captures the activity (e.g., sitting), nouns (e.g., woman, chair, office) related to the roles (Agent, Item, Place), and the bounding boxes of the objects. The caption template is “An Agent sits on an Item at a Place”, so the image caption of the verb sitting is “A woman sits on a chair at an office”. SAM uses bounding boxes as prompts to generate the segmentation masks. According to application interaction, potential region indication methods (e.g., fingertip, head pose, laser pointer) can be used to specify a region of interest, and the information can be reported to the user via bone-conducting earphones of wearable systems.

MateRobot is a wearable robotic system aimed at assisting people with visual impairments (PVI) by providing them with the ability to recognize various materials through visual cues. This technology represents a significant advancement in improving the autonomy and mobility of individuals with visual impairments, enabling them to interact more effectively with their environment.

The research demonstrated that it is feasible for this system to accurately identify material properties using advanced machine learning techniques. The ultimate goal is to enhance the functionality of wearable robotics by integrating material recognition capabilities, thereby enriching the sensory experiences of users.

• MateRobot utilizes a vision-based approach to identify different materials, which is essential for helping users understand their surroundings and make informed decisions.
• The system is designed to be user-friendly for PVI individuals, focusing on improving their daily life by providing real-time feedback about the materials they encounter.
• The research employs state-of-the-art image segmentation algorithms that increase the accuracy of material recognition.

We develop a mobility and navigational aid system for visually impaired persons, i.e. blind persons or persons with low vision. The system should enable visually impaired persons to navigate safely in an unknown terrain. It uses computer vision methods to detect landmarks, obstacles and free ground surface in front of the user. Acoustic as well as haptic interfaces (e.g. vibrating elements) are used to give information to the user. We have already created a prototype to this end and develop the system in a user-centered way, which is made possible by our tight cooperation with KIT Center for Digital Accessibility and Assistive Technolgoy (ACCESS∂KIT).

In the long term, we wish to allow visually impaired people to move around and to find their way safely on their own in a new environment, such as an unknown city or any other urban or rural environment. In the short term, our goal is to build a mobile system that helps visually impaired students to safely explore our university campus.

Our goal is to build a mobile assistive system to support the mobility of visually impaired people in various ways:

• provide orientation and navigation information (like GPS navigation systems)
• warn user before obstacles, both on the ground as well as high/low hanging obstacles
• plan a route beforehand, i.e., before a user can reach it with the white cane, with respect to the current situation, e.g., detect accessible section and obstacles along the path
• provide additional information about "the scene", e.g., type of intersection, walkway, people or cyclists approaching or oncoming traffic
• create situation specific "modules", e.g., crossing an inaccessible road intersection (button location, traffic light detection, anti-veering), help with zebra crossings, locating building entries and many others.