Perceiving Graphical and Pictorial Information via Hearing and Touch

We explore the use of hearing and touch for conveying graphical and pictorial information to visually impaired people. The main idea is that the user actively explores a two-dimensional layout consisting of one or more objects with the finger on a touch screen. The objects are displayed via sounds and raised-dot tactile patterns. The finger acts as a pointing device and provides kinesthetic feedback. The touch screen is partitioned into regions, each representing an element of a visual scene or graphical display, and each modality a different aspect of the layout. A key element of our research is the use of spatial sound to facilitate the active exploration of the layout. We use the head-related transfer function for rendering sound directionality and variations of sound intensity or tempo for rendering proximity. Our research has addressed object shape and size perception, as well as the perception of a 2-D layout of simple objects (dots). We have also considered the rendering of a simple scene layout consisting of objects in a linear arrangement, each with a distinct tapping sound, which we compare to a "virtual cane." Subjective experiments with visually-blocked subjects demonstrate the effectiveness of the proposed approaches.
For event registration and more details, please visit : IEEE Signal Processing Society Santa Clara Valley Chapter

Speaker: Dr. Thrasyvoulos N. Pappas, Northwestern Univ., Lawrence Livermore National Labs