Interdisciplinary Computational Art

Gesture / Language / Mirror

Dr. David Gruber & Dr. Daniel Howe — City University of Hong Kong

'Gesture / Language / Mirror' takes 'mirror neuron' (MN) findings from the neurosciences as inspiration for a new algorithm that matches gestures across videos of actors performing group therapy transcripts. The algorithm detects changes in hand and body position, causing the videos to 'jump' to the closest matching gesture. Videos run simultaneously while one screen acts as the 'driver' and the others search for gesture matches — creating a fragmented narrative driven by gesture and posture instead of chronology.

MNs have been employed to elaborate and support 'mirror therapy' in group psychotherapy settings; accordingly, we use computation to investigate how neuroscience findings translate as everyday practices. 'Gesture / Language / Mirror' re-performs the uptake of MNs. The experience of the exhibit tempts viewers to consider the role of gesture in communication, to look for the many subtle differences (in style, tone, mood) of 'mirrored' gestures, to realize the significance of verbal context while viewing the videos, and to then reflect on the MN literature in the neurosciences.