Employing the Hand as an Interface Device

Abstract

We propose algorithms and applications for using the hand as an interface device in virtual and physical spaces. In virtual drawing, by tracking the hand in 3-D and estimating a virtual plane in space, the intended drawing of user is recognized. In a virtual marble game, the instantaneous orientation of the hand is simulated to render a graphical scene of the game board. Real-time visual feedback allows the user to navigate a virtual ball in a maze. In 3-D model construction, the system tracks the hand motion in space while the user is traversing edges of a physical object. The object is then rendered virtually by the computer. These applications involve estimating the 3-D absolute position and/or orientation of the hand in space. We propose parametric modelling of the central region of the hand to extract this information. A stereo camera is used to first build a preliminary disparity map of the hand. Then, the best fitting plane to the disparity points is computed using robust estimation. The 3-D hand plane is calculated based on the disparity plane and the position and orientation parameters of the hand. Tracking the hand region over a sequence of frames and coping with noise using robust modelling of the hand motion enables estimating the trajectory of the hand in space. The algorithms are real-time and experiments are presented to demonstrate the proposed applications of using the hand as an interface device.