**Exploring Upper Limb Segmentation with Deep Learning for Augmented Virtuality** Monica Gruosso, Nicola Capece, Ugo Erra Department of Mathematics, Computer Science and Economics, University of Basilicata, Potenza, Italy 85100 monica.gruosso@unibas.it, nicola.capece@unibas.it, ugo.erra@unibas.it ![](teaser.png width="100%") _We propose a deep learning-based approach to enhance the user's sense of presence in virtual environments (VEs) allowing users to see their upper limbs instead of virtual hands (bottom panel). Hand and arms are captured using a common RGB camera positioned on the Head Mounted Display (HMD) or user's head. Then, images are processed by our upper limb segmentation network, which proved to be robust to different skin tones, lighting conditions, clothes, and occlusions (top panel). Finally, the segmented human limbs are visualized into the VE, while the interaction is allowed via a Leap Motion controller._ Abstract =============================================================================== Sense of presence, immersion, and body ownership are among the main challenges concerning Virtual Reality (VR) and freehand-based interaction methods. Through specific hand tracking devices, freehand-based methods can allow users to use their hands for VE interaction. To visualize and make easy the freehand methods, recent approaches take advantage of 3D meshes to represent the user's hands in VE. However, this can reduce user immersion due to their unnatural correspondence with the real hands. We propose an augmented virtuality (AV) pipeline allows users to visualize their limbs in VE to overcome this limit. In particular, they were captured by a single monocular RGB camera placed in an egocentric perspective, segmented using a deep convolutional neural network (CNN), and streamed in the VE. In addition, hands were tracked through a Leap Motion controller to allow user interaction. We introduced two case studies as a preliminary investigation for this approach. Finally, both quantitative and qualitative evaluations of the CNN results were provided and highlighted the effectiveness of the proposed CNN achieving remarkable results in several real-life unconstrained scenarios. Video ===============================================================================  BibTeX =============================================================================== ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @INPROCEEDINGS{"10.2312:stag.20211483", title = "Exploring Upper Limb Segmentation with Deep Learning for Augmented Virtuality", author = "Gruosso, Monica and Capece, Nicola and Erra, Ugo", booktitle = "Smart Tools and Apps for Graphics - Eurographics Italian Chapter Conference", year = "2021", publisher = "The Eurographics Association", doi = "10.2312/stag.20211483", } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Resources =============================================================================== We will release our dataset for encouraging future research on egocentric upper limb segmentation. Please send an email to monica.gruosso@unibas.it or nicola.capece@unibas.it if you need it for academic research and non-commercial purposes. Before requesting our data, please verify that you understand and agree to comply with the following: - This data may ONLY be used for non-commercial uses (This also means that it cannot be used to train models for commercial use). - You may NOT redistribute the dataset. This includes posting it on a website or sending it to others. - You may include images from our dataset in academic papers. - Any publications utilizing this dataset have to reference our paper, [EDSH](https://openaccess.thecvf.com/content_cvpr_2013/papers/Li_Pixel-Level_Hand_Detection_2013_CVPR_paper.pdf) and [TEgO](https://dl.acm.org/doi/10.1145/3290605.3300566) papers. - These restrictions include not just the images in their current form but any images created from these images (i.e. "derivative" images). - Models trained using our data may only be distributed (posted on the internet or given to others) under the condition that the model can only be used for non-commercial uses. For more information about original EDSH and TEgO data, please visit the following pages: - EDSH web page: http://www.cs.cmu.edu/~kkitani/datasets/ - TEgO web page: https://iamlabumd.github.io/tego/ Acknowledgments =============================================================================== The authors thank NVIDIA's Academic Research Team for providing the Titan Xp cards under the Hardware Donation Program.