An Analysis of Visual and Morphosyntactic Cues in Biased Polar Questions in Dutch
Elynn Louise Weijland

Abstract:
This thesis project analysed the visual cues that mark different types of polar questions in Dutch. From previous multimodal studies, we know that visual cues, such as eyebrow movements and head tilts, may accompany spoken language questions [Nota et al., 2021, 2023, Zygis et al., 2017, da Silva Miranda et al., 2020, Miranda et al., 2021]. Similarly, preceding research has shown that visual cues play a significant role in sign languages [Baker et al., 2016]. Such cues, also referred to as non-manual markers (NMM), can play a role at the morphological, phonological, syntactic and pragmatic level [Pfau and Quer, 2010]. A recent development in sign language research concerns the quantitative studies of NMM in polar questions. Particularly, Esselink [2022] examined the NMM that mark biased polar questions in Sign Language of the Netherlands (NGT). It was reported that different types of polar question in NGT are marked with different combinations of NMM. For instance, these involve frowned eyebrows, in combination with squinted eyes, or raised eyebrows and widened eyes, which do not occur in contexts in which positive prior belief is contradicted with negative contextual evidence.
  Spoken languages are multimodal: auditory cues, involving speech, are combined with visual cues to relay communicative intention. It is well-known that in Dutch, prosodic patterns and differences in word order mark polar questions [Englert, 2010, Borràs-Comes et al., 2014, Gaasbeek, 2023]. However, the manner in which spoken Dutch employs visual cues as a means to mark polar questions is relatively understudied.

The current project had three objectives: identifying the question structures used most frequently in different types of Dutch biased polar questions, obtaining the most prototypical facial expressions marking these different types of questions, and comparing these results to those found in NGT [Esselink, 2022].

The data that was obtained for this project was elicited by means of an experiment in which native Dutch speakers interacted with confederates in small role-plays. This experimental design closely followed the design by [Oomen and Roelofsen, 2023a, Esselink, 2022], in order to be able to compare results between NGT and Dutch.
  During the experiment, participants were recorded by three cameras, as well as one 3D depth camera using the Live Link Face software [Epic Games, 2023]. This camera measured the participants’ activity of 61 facial landmarks, also referred to as blend shapes, and assigns them a value between 0 and 1 (indicating a low and high level of activity, respectively).

The data was analysed in three ways. Unfortunately, due to the scope of this project, only a preliminary analysis was performed. First, the video data was manually annotated, using the ELAN software [ELAN, 2023]. Both the most prominent visual cues and the used question structure were annotated. Furthermore, two methods of quantitative data analysis were employed on the 3D data set: the temporal progression of the blend shape data was visualised and the HDBScan clustering algorithm was implemented. The latter provided us with the most prototypical combinations of facial features.

The five most prototypical facial expressions marking Dutch questions involve: 
[1] Raised eyebrows and wide eyes (occurring mostly in situations where positive prior belief is later contradicted with negative evidence)
[2] Frowned eyebrows and squinted eyes (found frequently in situations where neutral contextual evidence is provided) 
[3] Squinted cheeks, squinted eyes and a sneered nose 
[4] Simultaneous squinted and wide eyes 
[5] The neutral facial expression 

The first two of these were also found to be question markers in NGT [Esselink, 2022], however, this was not the case for the latter three. Furthermore, the pattern of engagement of the corresponding features does not always resemble the patterns found in NGT.
  Lastly, future avenues of research are discussed in detail, regarding changes in the experimental setup, pre-processing and data analysis.