December 13th, 2017

Increasing the Expressive Power of Gesture-based Interaction on Mobile Devices

Thanks to the interactive high-resolution touchscreen, today’s mobile interfaces are completely touch-based. The simplest form of touch-based interaction is tapping on menus or icons, that borrows the point-and-click metaphor in graphical user interface (GUI). The users can also swipe, pinch, or other complex gestures that involve several strokes or multi touch.

In general, a gesture-based interaction is a communication channel between the users and the system. It starts with the user transmitting a message by drawing a gesture on the touchscreen device. Hence, the gesture encodes the information that the user wants to communicate. The device then decodes the gesture into the intended message, that can express different messages, such as commands or text. The capacity of the communication channel, i.e. the expressive power of a gesture-based
interaction, depends on the number of different gestures and how they can express varying concepts.

Today’s smartphones offer a wide variety of functions, from communication, games, information consumption, office support, even art and creativity support. As a consequence, the command space becomes larger as well. To access a larger command space, the users are often forced to go through multiple steps e.g. searching the item in a menu, making the interaction cumbersome and inefficient. More complex gestures (e.g. different shapes, multi-touch, etc.) that offer a powerful way to access information quickly as well as to perform a command more efficiently. However, they are more difficult to learn and control. The more gestures in the gesture set, the harder it is for the system to accurately recognize the gestures.