SLR Papers

A large collection of papers on sign language recognition-translation and SLR datasets.

Table of Content

• Papers

  • HMM
  • CNN+RNN
  • 3D CNN
  • GCN
  • Others
  • Multi-modal
  • Sign Language Translation (SLT)
  • Surveys

• Datasets

• SLR Benchmarks

Papers

HMM

1. Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos IEEE-TPAMI2019paper [code]

2. Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs CVPR2017 paper code

3. Online Early-Late Fusion Based on Adaptive HMM for Sign Language Recognition TOMM2017 paper code

4. Chinese sign language recognition with adaptive HMM ICME2016 paper code

5. Sign language recognition based on adaptive HMMS with data augmentation ICIP2016 paper code

6. Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled CVPR2016 paper code

7. Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition BMVC2016 paper code

8. Continuous sign language recognition using level building based on fast hidden Markov model Pattern Recognit.Lett.2016 paper code

9. Sign Transition Modeling and a Scalable Solution to Continuous Sign Language Recognition for Real-World Applications TACCESS2016 paper code

10. A Threshold-based HMM-DTW Approach for Continuous Sign Language Recognition ICIMCS2014 paper code

11. Improving Continuous Sign Language Recognition: Speech Recognition Techniques and System Design SLPAT2013 paper code

12. Using Viseme Recognition to Improve a Sign Language Translation System IWSLT2013 paper code

13. Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition CVPRW2011 paper code

14. Speech Recognition Techniques for a Sign Language Recognition System INTERSPEECH2007 paper code

15. Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models TSMC2007 paper code

16. Real-time American sign language recognition using desk and wearable computer based video TPAMI1998 paper code

2D-CNN

1. A Comprehensive Study on Sign Language Recognition Methods Arxiv2020 paper code
2. A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training IEEE-TMM2019 paper [code]
3. Fingerspelling recognition in the wild with iterative visual attention ICCV2019 paper code
4. Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization CVPR2017 paper code
5. SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition ICCV2017 paper code

3D-CNN

1. Iterative Alignment Network for Continuous Sign Language CVPR2019 paper code

2. Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages? BMVC2019 paper code

3. Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks ACCESS2019 paper code

4. Dense Temporal Convolution Network for Sign Language Translation IJCAI2019 paper code

5. Thai Sign Language Recognition Using 3D Convolutional Neural Networks ICCCM2019 paper code

6. Sign Language Recognition Analysis using Multimodal Data DSAA2019 paper code

7. SF-Net: Structured Feature Network for Continuous Sign Language Recognition ArXiv2019 paper code

8. Video-based sign language recognition without temporal segmentation AAAI2018 paper code

9. Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition ICPR2016 paper code

10. Hand Gesture Recognition with 3D Convolutional Neural Networks CVPRW2015 paper code

11. Sign Language Recognition using 3D convolutional neural networks ICME2015 paper code

12. Dynamic Pseudo Label Decoding for Continuous Sign Language Recognition ICME2019 paper

13. Iterative Alignment Network for Continuous Sign Language Recognition CVPR2019 paper

14. Learning Spatiotemporal Features Using 3DCNN and Convolutional LSTM for Gesture Recognition ICCV2017 paper code

15. Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks CVPR2016 paper code

Skeleton-Pose based SLR

1. 3D Hands, Face and Body Extraction for Sign Language Recognition ECCV2020paper
2. Spatial-Temporal Graph Convolutional Networks for Sign Language Recognition ICANN2019 paper code
3. SIGN LANGUAGE RECOGNITION WITH LONG SHORT-TERM MEMORY ICIP2016 paper code
4. Sign Language Recognition and Translation with Kinect AFGR2013 paper code

Multi-modal

1. Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network Sensors2021 paper
2. Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-Latent Space IEEEACCESS2020 paper
3. Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition AAAI2020 paper

Sign Language Translation

1. Stochastic Transformer Networks With Linear Competing Units: Application To End-to-End SL Translation ICCV2021 paper

2. Sign Language Translation with Transformers ArXiv2020 paper code

3. Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation ArXiv2020 paper code

4. Neural Sign Language Translation CVPR2018 paper code

5. Hierarchical LSTM for Sign Language Translation AAAI2018 paper code

Others

1. Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison WACV2020 paper code

2. Human-like sign-language learning method using deep learning ETRI2018 paper code

3. Iterative Reference Driven Metric Learning for Signer Independent Isolated Sign Language Recognition. ECCV2016 paper code

4. Automatic Alignment of HamNoSys Subunits for Continuous Sign Language Recognition LREC2016 paper code

5. Curve Matching from the View of Manifold for Sign Language Recognition ACCV2014 paper code

6. Large-scale Learning of Sign Language by Watching TV (Using Cooccurrences). BMVC2013 paper code

7. Sign Language Recognition using Sequential Pattern Trees CVPR2012 paper code

8. Sign language recognition using sub-units JMLR2012 paper code

9. American Sign Language word recognition with a sensory glove using artificial neural networks Eng.Appl.Artif.Intell.2011 paper code

10. Learning sign language by watching TV (using weakly aligned subtitles) CVPR2009 paper code

Surveys

1. Quantitative survey of the state of the art in sign language recognition paper
2. Artificial Intelligence Technologies for Sign Language Sensors2021 paper

Datasets

Dataset	Language	Classes	Samples	Data Type	Language Level
CSL Dataset I	Chinese	500	125,000	Videos&Depth from Kinect	isolated
CSL Dataset II	Chinese	100	25,000	Videos&Depth from Kinect	continuous
RWTH-PHOENIX-Weather 2014	German	1,081	6,841	Videos	continuous
RWTH-PHOENIX-Weather 2014 T	German	1,066	8,257	Videos	continuous
ASLLVD	American	3,300	9,800	Videos(multiple angles)	isolated
ASLLVD-Skeleton	American	3,300	9,800	Skeleton	isolated
SIGNUM	German	450	33,210	Videos	continuous
DGS Kinect 40	German	40	3,000	Videos(multiple angles)	isolated
DEVISIGN-G	Chinese	36	432	Videos	isolated
DEVISIGN-D	Chinese	500	6,000	Videos	isolated
DEVISIGN-L	Chinese	2000	24,000	Videos	isolated
LSA64	Argentinian	64	3,200	Videos	isolated
GSL isol.	Greek	310	40,785	Videos&Depth from RealSense	isolated
GSL SD	Greek	310	10,290	Videos&Depth from RealSense	continuous
GSL SI	Greek	310	10,290	Videos&Depth from RealSense	continuous
IIITA -ROBITA	Indian	23	605	Videos	isolated
PSL Kinect	Polish	30	300	Videos&Depth from Kinect	isolated
PSL ToF	Polish	84	1,680	Videos&Depth from ToF camera	isolated
BUHMAP-DB	Turkish	8	440	Videos	isolated
LSE-Sign	Spanish	2,400	2,400	Videos	isolated
MS-ASL	American	1000	25,513	Videos	isolated
Purdue RVL-SLLL	American	39	546	Videos	isolated
RWTH-BOSTON-50	American	50	483	Videos(multiple angles)	isolated
RWTH-BOSTON-104	American	104	201	Videos(multiple angles)	continuous
RWTH-BOSTON-400	American	400	843	Videos	continuous
WLASL	American	2,000	21,083	Videos	isolated

SLR Benchmarks

Evaluation on Phoenix 2014 dataset

Method		Non-Manual	Manual features	Skeleton	Full-frame	Hands-frame	Motion-Opt.Flow	Validation	Test
CSLR		x	x					55.0	53.0
Align Hamnosys		x	x					49.6	48.2
1 Million Hands			x			x		47.1	45.1
SubUNets			x		x	x		40.8	40.7
Deep Sign			x			x		38.3	38.8
Staged Optimization						x		39.4	38.7
Without Segmentation			x		x			-	38.3
Parallel Temp. Encoder						x		38.1	38.3
Reinforcement Learning						x		38.0	38.3
Temporal Fusion						x		37.9	37.8
Cnn-temp-rnn								37.9	37.6
Dilated Convolutions					x			38.0	37.3
Align-iOpt					x			37.1	36.7
Dense Temporal Conv					x			35.9	36.5
SF-Net					x			35.6	34.9
DPD					x			35.6	34.5
Hybrid CNN-HMM						x		31.6	32.5
Fully-Inception Networks					x			31.7	31.3
GoogLeNet+TConvs					x			28.9	29.1
Phonological Subunits				x				-	28.1
Re-Sign					x			27.1	26.8
Multi-Stream CNN-HMMs		x	x		x	x		26.0	26.0
Fully Conv Networks					x			24.6	24.6
Cnn-Temp-Rnn					x			23.8	24.4
Cnn-Temp-Rnn					x		x	23.1	22.9
Cross-Modal Alignment					x			23.9	24.0
ST Multi-Cue Network		x	x	x	x			21.1	20.7

Sign Language Production

A list of awesome work on Sign Language Production (SLP). It contains an extensive literature review of the field of deep-learning based SLP, with all relevant publications.

I am gathering these papers as literature for my PhD, and thought others may be interested. If you have any updates, please feel free to contribute or email me at b.saunders@surrey.ac.uk.

These are ordered by year, and I try to find the appropriate conference for each.

2020

Machine Translation from Spoken Language to Sign Language using Pre-trained Language Model as Encoder. Miyazaki, Morita, Sano. LREC2020 - https://www.aclweb.org/anthology/2020.signlang-1.23.pdf

Adversarial Training for Multi-Channel Sign Language Production. Saunders, Camgoz, Bowden. BMVC20 - https://www.bmvc2020-conference.com/assets/papers/0223.pdf

SignSynth : Data-Driven Sign Language Video Generation. Stoll, Hadfield, Bowden. ACV Workshop 20 - https://epubs.surrey.ac.uk/858501/

Can Everybody Sign Now ? Exploring Sign Language Video Generation from 2D Poses. Ventura, Duarta, Giro-i-Nieto. SLRTP Workshop 20. -https://slrtp.com/papers/extended_abstracts/SLRTP.EA.14.018.paper.pdf

Progressive Transformers for End-to-End Sign Language Production. Saunders, Camgoz, Bowden. ECCV20 - http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123560664.pdf

Skeleton-based Chinese Sign Language Recognition and Generation for Bidirectional Communication between Deaf and Hearing People. Xiao, Qin, Yin. Neural Networks 20 - https://www.sciencedirect.com/science/article/abs/pii/S089360802030040X

Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks. Stoll, Camgoz, Hadfield, Bowden. IJCV20 - https://link.springer.com/article/10.1007/s11263-019-01281-2

Neural Sign Language Synthesis: Words Are Our Glosses. Zelinka, Kanis. WACV20 - https://openaccess.thecvf.com/content_WACV_2020/papers/Zelinka_Neural_Sign_Language_Synthesis_Words_Are_Our_Glosses_WACV_2020_paper.pdf

2019

NN-Based Czech Sign Language Synthesis. Zelinka, Kanis, Salajka. SPECOM19 - https://link.springer.com/chapter/10.1007/978-3-030-26061-3_57

Cross-modal Neural Sign Language Translation. Duarte. ACM International Conference on Multimedia - https://imatge.upc.edu/web/sites/default/files/pub/cDuarteb.pdf

2018

Sign Language Production using Neural Machine Translation and Generative Adversarial Networks. Stoll, Camgoz, Hadfield, Bowden. BMVC18 - http://bmvc2018.org/contents/papers/0906.pdf