(1) to keep the focus of user action and interaction in the physical, not virtual space;
(2) to permit multiple, collaborating people to simultaneously engage in an interactive experience combining both real and virtual objects;
(3) to use computer-vision algorithms to identify activity in the space without requiring the participants to wear any special clothing or devices;
(4) to use narrative to constrain the perceptual recognition, and to use perceptual recognition to allow participants to drive the narrative;
(5) to create a truly immersive and interactive room environment.

We believe the KidsRoom is the first multi-person, full yautomated, interactive, narrative environment ever constructed using non-encumbering sensors. This paper describes the KidsRoom, the technology that makes it work, and the issues that were raised during the system's development.