CREATIV: Creating Co-Adaptive Human-Computer Partnerships

CREATIV explores how the concept of co-adaptation can revolutionize the design and use of interactive software. Co-adaptation is the parallel phenomenon in which users both adapt their behavior to the system’s constraints, learning its power and idiosyncrasies, and appropriate the system for their own needs, often using it in ways unintended by the system designer.

A key insight in designing for co-adaptation is that we can encapsulate interactions and treat them as first class objects, called interaction instruments. This lets us focus on the specific characteristics of how human users express their intentions, both learning from and controlling the system. By making instruments co-adaptive, we can radically change how people use interactive systems, providing incrementally learnable paths that offer users greater expressive power and mastery of their technology.

The project offers theoretical, technical and empirical contributions. CREATIV will develop a novel architecture and generative principles for creating co-adaptive instruments. The multi-disciplinary design team includes computer scientists, social scientists and designers as well as ‘extreme users’, creative professionals who push the limits of their technology. Using participatory design techniques, we will articulate the design space for co-adaptive instruments and build a series of prototypes. Evaluation activities include qualitative and quantitative studies, in the lab and in the field, to test hypotheses and assess the success of the prototypes.
The initial goal of the CREATIV project is to fundamentally improve the learning and expressive capabilities of advanced users of creative software, offering significantly enhanced methods for expressing and exploring their ideas. The ultimate goal is to radically transform interactive systems for everyone by creating a powerful and flexible partnership between human users and interactive technology.

Imagine living in a computational world centered around you as a user, a world where you need not learn and re-learn skills with every new application you use; where the system both learns from how you work and helps you evolve into a power user. We can accomplish this by treating interactions as first-class objects, by separating interaction from data and functionality, and by letting users control this interaction in flexible, reusable way.

The goal of the CREATIV project is to realize this vision for a large class of use cases that involve creative activities. Users will have their own personal set of interactive ‘instruments’ that they can use in a variety of contexts, instead of being forced to use the interaction techniques delivered with each application. For example, instead of struggling with each application’s color selector, a graphic designer will be able to use her own color picker instrument, customized with her preferred color palettes and color models. The same color instrument will work across applications, and even across devices, e.g. on her smartphone and tablet. Finally, instruments will record and provide interactive visualizations of their use patterns over time, enabling the graphic designer to discover, for example, which color she uses most often with a given color.

This vision relies partly on the basic concept of instrumental interaction [3], which my research group, |in|situ|, has explored at multiple levels, from the design of individual interaction techniques [1] to software architectures [8]. My goal in the CREATIV project is to significantly extend this idea by building on my concept of co-adaptation [11]. This is a natural phenomenon in which two parallel effects obtain: users adapt their behavior to the system’s constraints, learning both its power and its idiosyncrasies. Users also appropriate the system, often using it in ways unintended by the system designer, adapting it to meet immediate needs in the current context of use.

If we set out to create explicitly co-adaptive systems, we must consider two issues: how to support learning and how to support appropriation. We can envision a world in which users begin with simple instruments and over time, add power and expressivity to meet their individual needs. A child drawing on a tablet does not require the same level of color specification as a graphic designer: while the former will use simple predefined crayons, the latter will create her own, unique color palettes and visual styles, and will want to use them with any of her creative software tools. In fact, the child and the designer could work on the same drawing, each with their own tools. Of course, co-adaptive instruments must also adjust to the physical interaction device, whether a smartphone, a laptop computer or a novel tangible computing device. The challenge is to let users preserve their skills while accommodating real differences in the physical nature of each device and the changing context of use.

A key insight is that co-adaptive instruments have a life cycle as well as a power curve: Users may learn to use progressively more complex instruments, but they may also customize those instruments over time. Imagine a text instrument that lets you incrementally adjust text size, not just to current lighting conditions, but also to the state of your own eyes as you age. What about instruments that record your particular patterns of use, for example when organizing your files or email, or when surfing the web, providing you with interactive visualizations of your past activity, to help you reflect upon, find or reappropriate for future use? What about instruments that are not only incrementally learnable, but also allow you to create your own personal language of expression, that can be applied in different settings for different purposes, under your control?

Once co-adaptive instruments are pervasive, users will find it worthwhile to devote time and energy to learning them. Complex, sophisticated tools will become accessible because learned cognitive and sensori-motor skills will be maintained; users will be able to adapt these tools to their individual needs and preserve those adaptations over time. This vision moves beyond the holy grail of graphical interfaces of the past 25 years, i.e. to provide maximum ease-of-use and accessibility. Our focus instead is on creating incrementally learnable instruments that allow users to attain ever greater levels of power and expression. This is an extremely ambitious vision that could fundamentally change how we all interact with computers, potentially as significant as the introduction of the graphical user interface or the world-wide web. However, many research issues remain and for this vision to be adopted, we must establish concrete proofs of concept and a general architecture for implementing it.

CREATIV explores how the concept of co-adaptation can revolutionize the design of interactive software. Our multi-disciplinary approach includes creative professionals, ‘extreme users’ who push the limits of technology. They are often frustrated by the limits imposed by advanced creative software tools and seek novel forms of creative expression, constantly adapting their tools to meet their own creative needs. These users include artists and designers, as well as scientists, engineers and knowledge workers. They all struggle with, yet innovate in their everyday use of technology. Although one goal is to serve these users, our ultimate goal is to generalize the results to all users of computers, ideally achieving mass adoption.