**A Preliminary Investigation into a Deep Learning Implementation for Hand Tracking on Mobile Devices** Monica Gruosso^1, Nicola Capece^2, Ugo Erra^1, Francesco Angiolillo^1 ^1 Department of Mathematics, Computer Science and Economics, ^2 School of Engineering University of Basilicata, Potenza, Italy 85100 monica.gruosso@unibas.it, nicola.capece@unibas.it, ugo.erra@unibas.it, angiolillo.francesco@gmail.com Abstract =============================================================================== Hand tracking is an essential component of computer graphics and human-computer interaction applications. The use of RGB camera without specific hardware and sensors (e.g., depth cameras) allows developing solutions for a plethora of devices and platforms. Although various methods were proposed, hand tracking from a single RGB camera is still a challenging research area due to occlusions, complex backgrounds, and a variety of hand poses and gestures. We present a mobile application for 2D hand tracking from RGB images captured by the smartphone camera. The images are processed by a deep neural network, modified specifically to tackle this task and run on mobile devices, looking for a compromise between performance and computational time. Network output is used to show a 2D skeleton on the user's hand. We tested our system on several scenarios, showing an interactive hand tracking level and achieving promising results in the case of variable brightness and backgrounds, and small occlusions. Overview =============================================================================== ![](net_architecture.png width="100%")

This figure shows the network architecture at inference time, which is based on the first part of [RegNet](https://handtracker.mpi-inf.mpg.de/projects/GANeratedHands/content/GANeratedHands_CVPR2018.pdf). The input is a squared RGB-only image and the output is a tensor consisting of 2D joint heatmaps. All layers are drawn with blocks of the same size to simplify their graphical representation. Video =============================================================================== The video teaser of our work, which was presented at the $3rd$ _International Conference on Artificial Intelligence & Virtual Reality_ ([IEEE AIVR 2020](https://aivr.science.uu.nl/)).  BibTeX =============================================================================== ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @INPROCEEDINGS{"gruosso2020prel", title = "A Preliminary Investigation into a Deep Learning Implementation for Hand Tracking on Mobile Devices", author = "Gruosso, Monica and Capece, Nicola and Erra, Ugo and Angiolillo, Francesco", booktitle = "2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)", year = "2020", pages = "380-385", doi = "10.1109/AIVR50618.2020.00079", } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Resources =============================================================================== | Download | Description | |:--------:|:-----------:| | | Official pubblication: 3rd IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR 2020) | | | Poster presentation | Award =============================================================================== We are delighted and proud to have received the IEEE AIVR $2020$ Best Presentation Award. We enjoyed the conference a lot and really appreciated the comments and interest in our work. Thank you all! ![](award.png width="50%") ![](bestPresentationAward_aivr2020.png width="100%")