Passive Acoustic Tap Tracking Across Large Interactive Surfaces

Overview

We have designed, built, and fielded a system that locates the position of knocks and taps atop a large sheet of glass. Our current setup uses four contact piezoelectric pickups (donated by Panasonic) located near the sheet's corners to record the structural-acoustic wavefront coming from the knocks. A digital signal processor extracts relevant characteristics from these signals, such as amplitudes, frequency components, and differential timings, which are used to estimate the location of the hit and provide other parameters, including the rough accuracy of this estimate, the nature of each hit (e.g., knuckle knock, metal tap, or fist bang), and the strike intensity. As this system requires only simple hardware, it needs no special adaptation of the glass pane, and allows all transducers to be mounted on the inner surface, hence it is quite easy to deploy as a retrofit to existing windows. This opens many applications, such as an interactive storefront, with dynamic content controlled by knocks on the display window. We have thusfar run this system on two types of glass - 1/4 inch tempered glass (which achieves a knock position resolution of about s=2 cm across 1.5 meters of glass) and 1/2 inch shatterproof glass (which achieves position resolutions ranging between s=3-4 cm across 2 meters of glass). Using an Analog Devices ADSP401 for realtime signal analysis results in a latency of only 60 ms per hit, so the response is quite rapid.

Although the position resolution is somewhat more coarse, the immediate benefits of this approach over touch sensitive interfaces include the large size of the active surface, low cost, intrinsic simplicity, optical transparency, high robustness, and the fact that there's no need to mount any hardware outside the window.

We first concieved this project was as a tactile interface for a virtual fish tank, but this was never realized. The first workable system in this family stemmed from a collaboration between Joe Paradiso and the the Tangible Media Group for the ball impact tracker with the Ping Pong Plus interactive ping pong table. Tracking the hits of hands on sheets of glass is much harder, however, because of the highly dispersive nature of bending waves that propagate through the glass and the poorly defined transients that can vary considerably knock-to-knock. As our techniques were progressively refined, the performance of the system improved. We have deployed five such interactive window systems in installations operating for the general public, one at the Ars Electronica Center in Linz Austria, one at the Kitchen Gallery in Manhattan, another at the American Greetings store near Rockefeller Plaza (also in Manhattan), another at the SIGGRAPH 2002 Conference in San Antonio, TX, and one at Motorola in Plantation, Florida. More detail on these installations are linked below:

Installations

Early Development and Tests at the MIT Media Lab (1999-2000)

The Knock-Driven Browser at the MIT Media Lab (2001)

The Responsive Window at the Ars Electronica Center, Linz Austria (2001-present)

The TelephoneStory Installation at the ID/Entity Exhibit (MIT Media Lab & Kitchen Gallery NYC, 2001-2002)

Interactive Storefront Window at Rockefeller Center (2001-2002)

Interactive Window System at SIGGRAPH 2002 in San Antonio, TX

The Innovation Corner at Motorola's iDEN facility in Plantation FL (2003-present)

Publications about this system

  A paper presented on the system at the IEEE Sensors 2002 Conference
Passive Acoustic Sensing for Tracking Knocks Atop Large Interactive Displays, Joseph A. Paradiso, Che King Leo, Nisha Checka, Kaijen Hsiao. In the 2002 Proceedings of the 2002 IEEE International Conference on Sensors, Volume 1, Orlando, Florida, June 11-14, 2002, pp. 521-527.

A short paper presented at CHI2002
Passive Acoustic Knock Tracking for Interactive Windows, Joseph A. Paradiso, Che King Leo, Nisha Checka, Kaijen Hsiao. In the Proceedings of the ACM CHI2002 Conference, Minneapolis, MN, Extended Abstracts, April 20-25, 2002, pp. 732-733.

A paper on interactive surfaces describing the early system in the IBM Systems Journal
Sensor Systems for Interactive Surfaces, J. Paradiso, K. Hsiao, J. Strickon, J. Lifton, and A. Adler, IBM Systems Journal, Volume 39, Nos. 3 & 4, October 2000, pp. 892-914.

A brief writeup for the Ars Electronica 2002 Proceedings
Responsive Window, Joseph Paradiso. Takeover - Proc. of Ars Electronica 2001, Springer Vienna/New York, 2001, pp. 261-263.

Description of the Interactive Window installation at SIGGRAPH 2002
The Interactive Window, Joseph A. Paradiso, Che King Leo, Nicholas Yu, Marc Downie. In SIGGRAPH 2002, Conference Abstracts and Applications, ACM Press, 2002, p. 79.

A brief article on the Rockefeller Center tests that appeared in USA Today

Student Theses relevant to this project:

Contact and Free-Gesture Tracking for Large Interactive Surfaces, Che King Leo, MENG Thesis - MIT Department of EECS and MIT Media Lab, June 2002.

A System for Tracking and Characterizing Acoustic Impacts on Large Interactive Surfaces, Nisha Checka, M.Eng Thesis, MIT Department of EECS and MIT Media Lab, May 2001.