Authors
Natalie Hollain
Martha Larson
Floris Roelofsen
Date (dd-mm-yyyy)
2023
Title
Analyzing the Potential of Linguistic Features for Sign Spotting
Subtitle
A Look at Approximative Features
Publication Year
2023
Number of pages
10
Publisher
European Association for Machine Translation
Document type
Conference contribution
Abstract

Sign language processing is the field of research that aims to recognize, retrieve, and spot signs in videos. Various approaches have been developed, varying in whether they use linguistic features and whether they use landmark detection tools or not. Incorporating linguistics holds promise for improving sign language processing in terms of performance, generalizability, and explainability. This paper focuses on the task of sign spotting and aims to expand on the approximative linguistic features that have been used in previous work, and to understand when linguistic features deliver an improvement over landmark features. We detect landmarks with Mediapipe and extract linguistically relevant features from them, including handshape, orientation, location, and movement. We compare a sign spotting model using linguistic features with a model operating on landmarks directly, finding that the approximate linguistic features tested in this paper capture some aspects of signs better than the landmark features, while they are worse for others.

Permalink
https://hdl.handle.net/11245.1/d2365daa-d577-49bc-a7aa-831b7aa70fb8