Humans are a social species and evolution has equipped them with a unique capacity to navigate the social world. Apart from using spoken language, nonverbal cues play an essential part in achieving successful and harmonious social interactions. Yet, only recently have most psychiatric disorders begun to be conceptualized as “disorders of social interactions”. Personality disorders, schizophrenia, anxiety, depression and even neurodevelopmental disorders like autism are strongly coupled with impairments in the perception, interpretation and/or production of nonverbal cues. Analysing the dynamics of typical and atypical social interactions is therefore a natural and well-fitting means that may help to develop and enhance automatic psychiatric diagnosis and treatment tools. However, it is a major challenge to design accessible objective computational approaches that can tackle the complexity of naturalistic interpersonal behaviour. In this special session, we focus on challenges from two perspectives.

First, the rules of social interaction are largely unknown; however, most of the proposed methodological approaches to quantify typical and atypical nonverbal behaviour in social interactions rely on a set of predefined assumptions. Considering most of the work in social psychology focuses on microanalysis of vast amounts of videotaped data, data-driven approaches leveraging large datasets and power of deep learning are increasingly
needed, as they can automatically uncover the patterns of nonverbal interaction without relying on predefined rules in contrast to hypothesis-driven approaches. Though this poses the challenge of manually annotating large-scale datasets. We propose to shift focus from supervised learning settings to unsupervised, weakly supervised or self-supervised learning settings to provide new perspectives on social interaction analysis.

Second, most computer scientists have focused more on developing novel methods and improving performance, rather than investigating the usefulness of the developed techniques as a clinical tool in real-world applications. On the other hand, psychologists have been limited in terms of the interpretability, safety and reliability of the methodologies in decision-making, which are made available to them. Investigating explainable
black-box models are of paramount importance to provide meaningful insights on the understanding of social interactions. Some of these explainable methods are already finding applications in medical diagnosis – an area yet to be explored for psychiatric disorders.

The key aim of this multidisciplinary special session is to unite the power of computer scientists and social psychologists to discuss cutting edge research and innovative ideas for investigating data-driven, supervision-free or explainable methods to model interpersonal dynamics in both typical and atypical individuals (i.e., psychiatric disorders). More specifically, this special session sets out to put forward opportunities and
challenges for learning the rules of dyadic interactions or small group interactions from large amounts of video data or other modalities, without extensive use of manual supervision or prior assumptions, while encouraging the design of interpretable, safe and reliable techniques that can be adopted effectively in real-world clinical applications. We seek high-quality research papers on the following topics, or another topic closely relevant to
the special session theme:

  • Data-driven approaches to the analysis of nonverbal displays expressed within interpersonal context, including facial expressions, eye gaze and head movements, body postures and hand gestures, audio (e.g., turn taking, vocal outbursts, etc.), and the co-modelling of nonverbal and verbal cues;
  • Data-driven approaches to the modelling of interpersonal coordination such as convergence, synchrony or mimicry;
  • Automatic detection of abnormal social behaviour, namely, non-conforming patterns in nonverbal interaction;
  • Unsupervised/weakly supervised learning of social interaction, including representation learning, learning from interpersonal context, learning across data modalities, exploiting feature correlations, etc.;
  • Explainable deep models, ranging from extracting interpretable features and visualisation to analysing decision making processes and building interactive explanations and human-in-the-loop approaches;
  • Clinical applications (e.g., autism, depression, anxiety, etc.), including defining appropriate qualitative and quantitative evaluation methods;
  • Novel datasets comprising social interactions among typicals, psychiatric disorders, or mixed groups.


  • Oya Celiktutan, Department of Engineering, King’s College London, UK,
  • Alexandra L. Georgescu, Department of Psychology, King’s College London, UK,