'Gesture / Language / Mirror' is an interdisciplinary computational art project funded by City University of Hong Kong, led by Dr. David Gruber and Dr. Daniel Howe. The project takes 'mirror neuron' (MN) findings from the neurosciences as inspiration for a new algorithm that matches gestures across videos of actors performing group therapy transcripts. The algorithm detects changes in hand and body position, causing the videos to ‘jump’ to the closest matching gesture. Videos run simultaneously while one screen/video acts as the 'driver' and the others search for gesture matches. This creates a fragmented narrative driven by gesture and posture instead of chronology.

MNs have been employed to elaborate and support 'mirror therapy' in group psychotherapy settings; accordingly, we use computation to investigate how neuroscience findings translate as everyday practices. 'Gesture / Language / Mirror', thus, re-performs the uptake of MNs. The experience of the exhibit, we hope, tempts viewers to consider the role of gesture in communication, to look for the many subtle differences (in style, tone, mood) of 'mirrored' gestures, to realize the significance of (verbal) context while viewing the videos, and to then reflect on the MN literature in the neurosciences.

Gesture / Language / Mirror

Mirror neuron algorithm project.

The Performance - A Therapy Session

City University of Hong Kong, Sept. 2015
User chooses the 'driver' video and joins the therapy session

More Information