Royal Institution Christmas Lectures, London, 2008
Cisco Networkers, Anaheim, 2007
Interaction paper, co-authored with Randy Pausch and Steve Seitz, presented at ICMI
2002 in Pittsburgh, Pennsylvania.
The techniques we introduced in this paper have been featured in many places, including:
Spiderman 3 Premiere, 2007
Cinematrix Interactive Entertainment System
SIGGRAPH in 1991, Loren and Rachel Carpenter unveiled an
interactive entertainment system that allowed members of a
large audience to control an onscreen game using red and green
reflective paddles. In the spirit of this approach, we
developed a variety of techniques that enabled members of an
audience to participate, either cooperatively or
competitively, in shared entertainment experiences. These
techniques allow audiences with hundreds of people to control
onscreen activity by (1) leaning left and right in their
seats, (2) batting a beach ball while its shadow is used as a
pointing device, and (3) pointing laser pointers at the
screen. All of these techniques can be implemented with inexpensive, off the shelf hardware.
|Audience movement tracking allows audience members to control an onscreen game without the use of physical props. We position a camera at the front of the auditorium and point it at the audience. We then encourage the members of the audience to move in certain ways, and analyze the streaming video to control an onscreen game. For example, audience members can lean left or right in their chairs to steer a race car or move a paddle in the video game
Here are videos of a large audience playing Pong
(7 MB) and Pole
Position (11 MB), a classic car racing game.
technique uses pixel differencing to determine the amount of
motion in the crowd. In this "chase" game, the
left side of the audience controls the coyote and the right
side controls the roadrunner. The more that a side of
the audience moves, the faster their character runs. The
small image inserts on either side of the screen show the two
sides of the audience, and the bar graphs above them show the
level of audience activity on each side.
crowd activity of batting a beach ball before a concert
provided the inspiration for another interaction technique.
We project a game on the front screen of a movie theater, and
as the audience bats a beach ball into the air, the ball casts
a shadow on the screen. We point a camera at the screen and
use computer vision techniques to track the ball's shadow.
This tracking allows the audience to play interactive games
using the shadow of the ball as a cursor.
Here is a video
of the show (10 MB).
Command using beach ball shadow
The beach ball (top left) casts a
shadow (bottom right) which
acts as a cursor.
|It is not uncommon for members of a large movie theater audience to shine laser pointers at the movie screen before the film begins. By pointing a camera at the screen and tracking the dots, we can create compelling interactive entertainment experiences.
The red dots can be attached to a collaborative paint program
or a game that encourages maze navigation or rapid flocking of
laser points to particular locations. At right is an
example of a game in which the audience is encouraged to
uncover a series of hidden images using their laser pointers.
(13 MB) shows the game in action.
is another game that uses laser pointers as input
devices. The audience directs their laser points at the
hamsters to "whack" them. Here is a sample video
|I have also
used laser tracking to carry out live audience polls and
trivia. This is a video
showing a poll (9MB), and this
video (10 MB) shows a trivia question. The audience
members use their lasers to point at choices, and the
bar graphs continually update to show the audience's
preferences until the time limit is reached.
tracking works well with many different objects. The
game pictured at left allows the audience to raise and lower a
surface by "holding it up" with shadows. When
this game is shown in a large auditorium, the audience members
in the front row are given pool noodles (long, cylindrical Styrofoam
rods) that they hold in front of the screen to create
shadows. The top of each shadow is actually the control
point on a spline, and the audience attempts to modify the
surface so that the ball bounces into the target. This
video (11 MB) shows my research advisor giving it a try.
shadow tracking was in place, it was easy to build a simplified,
shadow-based version of the
Text Rain piece by Camille Utterback and Romy Achituv,
shown at Siggraph 2000. The
participants can hold up the falling cascade of letters using
their shadows. Here is a sample
video (23 MB).