**An implementation of the City Metaphor through Virtual Reality**
				
![Figure [teaser]: Code2City running: an example of 3D City built through the FindBugs metrics.](teaser.PNG width="100%")
</br></br>

Abstract
===============================================================================
_We describe a technique developed using C++ language and
Unreal Engine 4 that allows users to visualise software systems written in
object-oriented Java through virtual reality and using the city metaphor.
Our aim is to use virtual reality to visualise the metrics of types and
packages of a software system. In this report, we present a prototype called Code2City
that can be used in Virtual Reality mode or in a classical First-Person game 
mode and some implementation details. The ultimate goal will be to demonstrate 
that it is possible to use virtual reality to better understand software._				   

Overview
===============================================================================
Software visualization can benefit from recent display systems such as virtual reality (VR) devices. 
In this technical report, we present a technique to visualise object-oriented software systems written 
in Java through VR. This tool is an extension of the city metaphor proposed by Wettel et al.[#Wettel2008] and 
enables the user to visualise the characteristics of each class comprising the software system being 
analysed, by the user's full immersion in VR. The idea is to increase and further improve user interaction 
with the 3D scene, visualising other metrics such as the class dependencies. In fact, it would be possible 
to obtain a better perception of the structure of the analysed software system by selecting a class and 
determining the classes on which it depends. This technical report illustrates the design and the development 
of the tool for the 3D mode ad its extension to VR mode. The tool is a first step towards quantifying the 
use of VR for software analysis. The post-implementation step consists of user experimentation where a group 
of computer science students took in a controlled experiment  
on the use of the city metaphor displayed on a standard computer screen and in an immersive virtual reality.
The raw data and experimental materials from this experiment are available for download in the <a href="http://graphics.unibas.it/Code2City/index.md.html#resources"> Resources section</a>.

Pipeline Structure
===============================================================================
The proposed application is based on a well define pipeline: The first pipeline 
step consists of the Java software system analysis; The second pipeline step 
concerns city reconstruction and rendering of the 3D scene; The third step concerns 
user involvement in visualization and interaction with the City.

![Figure [pipeline]: Representation of our pipeline: The first step consists in 
java software system analysis and the second step consists of the construction 
of the City.](Pipeline.png width="100%")

Source Code Analysis
-------------------------------------------------------------------------------
Java software systems are composed of two main entities: **Packages** and 
**Types**. The analysis of the software system is performed using an 
<a href="https://www.eclipse.org/">Eclipse</a> plugin called 
<a href="https://wiki.eclipse.org/images/7/75/CodeProDatasheet.pdf">CodePro AnalytiX<sup>TM<sup></a>. 
This plugin seeks to improve the quality and security of the source code and 
helps developers to reduce the number of errors and security vulnerabilities. 
CodePro allows the desired metrics to be selected from the analysed software 
system code, generating an XML file containing the required information.

The first operation performed from CodePro AnalytiX is to take the directory 
containing the java source code of the software system to be analyzed. 
The second operation performed is the analysis and the extraction of the required 
metrics. Our source code analysis included for each type: the structure of its 
package through the fully qualified name; the number of the methods, the number 
of attributes. CodePro AnalytiX allows to export the obtained metrics in several 
formats such as HTML or XML, however, we chose the XML format because it represents 
a generic standard that may also be used in other application contexts.

![Figure [schema]: The positioning algorithm and the Tree-Type structure of the 
Java software system. The packages are the intermediate node and the types 
are the leaf.](Bin-packing.png width="100%")

City 3D Reconstruction and Rendering
-------------------------------------------------------------------------------
The XML file generated from CodePro AnalytiX is structured as a list of type, 
and the parameters of the XML sub-tags represent the metrics of the type. For 
this reason, is necessary to reconstruct the software system schema through a 
tree-type data structure. This operation is performed through our Code2City tool 
as shown in Figure [pipeline]. Tree-Type Output is used to compute the dimensions 
of the 3D objects that represent the packages of the software system. Such 
algorithm called <a href="https://codeincomplete.com/posts/bin-packing/">bin-packing</a> 
computes the dimension of block based on the objects that are inside it, optimising 
the layout to minimise the space occupied. The algorithm is executed recursively, 
starting from the leaves, to compute the dimension and positions in the 3D scene 
for all packages. The final step of our pipeline consists of building the 3D 
scene based on the position and the dimension of each package and type in the tree. 
Packages are shown as colourful in grey scale in a range from dark grey to light 
grey according to the nesting level of the package in the software system [#Wettel2007]. 
The height of the package 3D object is a constant value. Types are represented as 
blocks in blue scale, represented on a scale from 0 to 1, where 0 means dark blue and 1
means light blue. The colour value is given by the number of lines of code, which is 
composed of a certain type that has been normalized between 0 and 1, to display a type 
composed of a few lines of code with a darker colour and a type composed of many lines 
of code with a lighter colour. The width of the building is given from the number of 
attributes that belong to the type. This number is multiplied by a scaling factor 
to represent the city in proportion to the user. The final displayed parameter 
represents the height of each building, which is given from the number of methods 
belonging to the type. Also, this number is multiplied by a scaling factor to balance
the city and the user dimension.  

Visualization and Interaction Methods
-------------------------------------------------------------------------------
<!--Third person visualisation (descrivere i movimenti e i controller e tutto il resto)
fai riferimento anche al tutorial, parlare anche della VR puoi usare come riferimento anche
la parte finale the The VR System-->
<!--Code2City allows two visualization and interaction gaming styles: classical 
First-person 3D gaming and Virtual Reality gaming.-->
Code2City is a software application tool that allows two visualization and interaction gaming styles: 
classical First-person 3D gaming and Virtual Reality gaming. These two visualization modes are both 
eveloped using the same 3D engine and the user at the starting selects one of the two modes.

### 3D Mode

In 3D mode a standard CameraActor of unreal engine 4 was used to allow the user to view the 
3D scene. The user interacts with Code2City by using mouse and keyboard.

![Figure [interactionDev]: Description of the keyboard and mouse command for 
the interaction with Code2City in first-person gaming style.](keymouse.png width="100%")

<!--Infact, the user can fly over the 3D City, climb on top of the building, and look
at the city from the highest buildings. The user can also move and observe the
city from the perspective of a pedestrian who walks inside the 3D City as shown
in the video below. The 3D City dimenson is proportional to the size of the user.
Through the keyboard, the user can move in any direction. To this end, we applied
the WASD based video game control method. As shown in Figure [interactionDev] the
**W** and **S** keys control forward and backward movements, respectively. On the
other hand, the **A** and **D** keys control left and right starfing, respectively.
To move upwards, the user exploits the **space** key, while the **ctrl** key is used
to move downwards. The mouse allows rotation with respect the normal and tranverse axes.-->

The user can fly over the 3D City, climb on top of the building, and look at the city from 
the highest buildings. The user can also move and observe the city from the perspective of 
a pedestrian who walks inside the 3D City as shown in the video below. The 3D City dimension 
is proportional to the size of the user. Through the keyboard, the user can move in any 
direction using the WASD keys control method used in any first-person shooter video game. 
As shown in Figure [interactionDev], the **W** and **S** keys control forward and backward movements, respectively. 
On the other hand, the **A** and **D** keys control left and right strafing, respectively. 
To move upwards, the user exploits the **Space** key, while the **Ctrl** key is used to move downwards. 
The mouse allows rotation with respect to the normal and transverse axes.

<!--To interact with the 3D objects, the user can use the **Q** key to show the viewfinder
and select a desired object. When the selected object is a building, it changes color
becoming yellow and further information on the corresponding object is shown. If the 
object correspond to a type, the system shows: type name, number of methods, number 
of properties, number of code lines, and package name to which it belongs. If the 
object corresponds to a package, its full name and the contained classes are shown to
the user. Code2City allows searching the types of a subject program by specifying their
names. Indeed, the users wirte down a search string that shold match either name of the 
type or its parts. In particular, through the numeric key, the key **Num 1** the textbox is shown
and by pressing the **Enter** key, the search by name is performed. In a similar way is
possible to search a word in the properties and in a methods of the types by using the
key **Num 2**. Types that respect the search string are highlighted in red. Code2City also 
allows searching types by callers. This feature allows identifying the type, which perform 
calls to methods of the type provided by the user. The methods are provided by using the 
key **Num 3** that shows the textbox and the **Enter** key perorm the search. Types that 
respect the search criterion are also highlighted in red.-->

To interact with the 3D objects, the user uses the **Q** key to show the viewfinder and select 
the desired object. When the selected object is a building, it changes color becoming yellow 
and further information on the corresponding object is shown. If the object corresponds to a 
type, the system shows: type name, number of methods, number of properties, number of code 
lines, and package name to which it belongs. If the object corresponds to a package, its full 
name and the contained classes are shown to the user. Code2City allows searching the types of 
a subject program by specifying their names. Indeed, the user writes a search string that 
should match either name of the type or its parts. In particular, through the numeric key, 
the key **Num 1** the textbox is shown and by pressing the **Enter** key, the search by name is performed. 
In a similar way is possible to search a word in the properties and in a method of the types 
by using the key **Num 2**. Types that respect the search string are highlighted in red. Code2City 
also allows searching types by callers. This feature allows identifying the type, which performs 
calls to methods of the type provided by the user. The methods are provided by using the key 
*Num 3* that shows the textbox and the **Enter** key perform the search. Types that respect the 
search criterion are also highlighted in red.

Code2City supports the identification of possible design problems using different colors
to distinguish among the following smells: _brain classes_ are shown in light green by
pressing the key **Num 7**, _data classes_ are shown in heavenly by pressing key **Num 8**
and finally the _god classes_ are shown in violet by pressing **Num 9**.

### VR Mode

<!--In VR mode, Code2City implements the same features as in 3D mode. That is,
it provides the same support of the 3D mode when carrying out the experiment tasks.
The interaction and visualization modes in VR mode are different from the 
3D mode. In fact, the user is provided with an Xbox One wireless controller and an
Oculus Rift Head Mounted Display (HMD) that allow the immersion in the 3D environment.-->

In VR mode, Code2City implements the same features in the 3D version of the tool. That is, 
it provides the same support of the 3D mode when carrying out the experiment tasks. 
The interaction and visualization modes in VR mode are different from the 3D mode. 
In fact, the user is provided with an Xbox One wireless controller and an Oculus Rift 
Head Mounted Display (HMD) that allow the immersion in the 3D environment.

![Figure [controllerDev]: Description of the Xbox One wireless controller command for 
the interaction with Code2City in virtual reality gaming style.](Controller.png width="100%")

<!--The user interacts with the Code2City in VR through the head movements that allow rotations 
with respect to the normal and tranverse axes. Differently from the mouse, HMD also allows
a rotation with respect to the longitudinal axis. The user can select the virtual object 
using the direction in front of his head and the **A** controller button.-->

The user interacts with the Code2City in VR through the head movements that allow rotations 
with respect to the normal and transverse axes. Differently, from the mouse, HMD also allows 
a rotation with respect to the longitudinal axis. The user can select the virtual object using 
the direction in front of his head and the **A** controller button.


![Figure [axes]: Six degrees of freedom: forward/back, up/down, left/right, normal, 
transverse and longitudinal axes.](axes.png width="50%")
In this way, the user has the feeling of being really in a city. The user can change her 
movement speed by pressing the **Right Trigger** controller button. Since a physical keyboard 
is not present searching types are supported through the virtual keyboard, which can be 
activated and deactivated through the **Left Bumper** controller button. To interact with the 
virtual keyboard, the user exploits the **Right Stick** of the controller as shown in Figure[controllerDev]. 
To speed up the writing time for text queries, Code2City provides an automatic completion feature. 
Also in VR mode, it is possible to perform the same features of the first-person game style.

<!--In this way, the user has the feeling of being really in a city. The user can change her
movement speed by pressing the **Right Trigger** controller button. Since a physical 
keyboard is not present seaching types are supported through the virtual keyboard, which
can be activated and deactivated through **Left Bumper** controller button. To interact
with the virtual keyboard, the user exploits the **Right Stick** of the controller as shown
in Figure[controllerDev]. To speed up the writing time for text queries, Code2City provides 
an automatic completion feature. Also in VR mode it is possible to perform the same features
of the first-person game style.-->


<!--Quando parlo della VR posso dire che la versione descritta nel paper AVR è una prima versione
e che nella versione successiva abbiamo iniziato ad usare il controller classico della XBox One-->

Implementation Details
===============================================================================
<!--Fai riferimento al background, quando si inizia a parlare di UE4 e vedi la seconda parte 
di The VR System-->

<!--To perform the 3D City construction and rendering we developed an application 
called Code2City based on well-known videogames engine <a href="www.unrealengine.com">Unreal Engine 4</a> (version 4.15.3).
We used this game engine because it provides a good trade-off between rendering of the scenes,
ease to use, and performance of immersive virtual reality. In particular we used
Windows 10 Pro on a computer equipped with 16 GB of RAM, Intel Core  i7-3820 (3.6 GHz)
as CPU and NVIDIA Titan XP as GPU. We obtained as rendering performance 45 frames
per seconds in VR mode and 60 frames per seconds in First-Person mode on FindBugs
software system visualizing. Unreal Engine 4 allows 3D applications to be developed using two different styles of 
programming: **Blueprints Visual Scripting** and **C++ programming language**. 
These programming styles can be mixed to simplify the development process and for 
this reason, we used both styles. The C++ programming language was used to read 
the Tree-Type structure and create the mesh that belongs to the Code2City. In 
particular, the XML file was read using the <a href="https://www.boost.org">Boost C++ libraries</a> 
which encapsulate the <a href="http://rapidxml.sourceforge.net/">Rapidxml library</a> 
as default parse with a higher level interface and easy to use and to reconstruct 
the XML schema in the C++ data structure.-->

In order to design the 3D City construction and rendering, we developed an application 
called Code2City based on well-known videogames engine <a href="www.unrealengine.com">Unreal Engine 4</a> (version 4.15.3). 
We used this game engine because it provides a good trade-off between the rendering of 
the scenes, easy to use, and performance of immersive virtual reality. In particular, 
we used Windows 10 Pro on a computer equipped with 16 GB of RAM, Intel Core i7-3820 (3.6 GHz) 
as CPU and NVIDIA Titan XP as GPU. We obtained as rendering performance 45 frames per seconds 
in VR mode and 60 frames per seconds in First-Person mode on FindBugs software system visualizing. 
Unreal Engine 4 allows 3D applications to be developed using two different styles of programming: 
**Blueprints Visual Scripting** and **C++ programming language**. These programming styles can be mixed 
to simplify the development process and for this reason, we used both styles. The C++ programming 
language was used to read the Tree-Type structure and create the mesh that belongs to the Code2City. 
In particular, the XML file was read using the <a href="https://www.boost.org">Boost C++ libraries</a> which encapsulate the 
<a href="http://rapidxml.sourceforge.net/">Rapidxml library</a> as default parse with a higher level interface and easy to use and to reconstruct the XML 
schema in the C++ data structure.


![Figure [blueprints]: A basic example of Blueprints Visual Scripting. In this 
script, the basic colour of the dynamic material was set. This colour is recomputed 
and changed at runtime.](blueprints.PNG width="100%")

The Blueprints Visual Scripting is based on the concept of using a node-based 
interface to create gameplay elements. We used this programming style to develop the
user's interactions and some rendering components: the render materials and 
lightings. In the Unreal Engine 4 editor, we configured a basic scene with one 
directional light, shadows and an unreal engine object called SkyLight that simulates the sky 
and its lights, reflections, atmosphere and the clouds. These components allow improving 
the photorealism of the scene. In particular, the SkyLight is useful for viewing well
even in shadow zones, and the second is useful for displaying the shadows of buildings.
When the application is running, the basic scene is merged with the 3D objects of the 
City and the shadows are recomputed based on the buildings positions. The colour of 
buildings is handled using a dynamic material, i.e. an instanced material that can 
be calculated at runtime (see Figure [blueprints]).

![Figure [editor]: Figure shows the scene graph after the city was built and rendered.](editor.PNG width="100%")

To interact with the scene we used ray casting extensively [#Stoakley1995] to locate 
and select objects in the 3D scene. 

In a first version of Code2City [#Capece2017] we
use the HMD called <a href="https://www.vive.com/">HTC Vive </a>
and its controllers. In this version a ray is shot orthogonally to the controllers position. 
We proposed another version of Code2City compatible with the HMD called <a href="https://www.oculus.com/">Oculus Rift</a> 
that is equipped with an Xbox One controller (see Figure [controllerDev]). In this version
a ray is shot orthogonally to the user head position inside the scene. Ray is represented by a 
vector of the constant module and allows the user to point the controller or the gaze towards 
the desired object. When the ray  hits a building, it changes the colour of the building, 
exploiting the dynamic variation of the material at runtime. The aim is to give the user 
the illusion of the selection of the building. After the building was selected, the user 
can view detailed information on metrics belonging to the represented type. To shows this 
information as a text in the virtual scene we used an unreal engine object called 
_UTextRenderComponent_. The text is attached to the user camera called also _CameraActor_ 
and follows the user movement inside the scene. After the building has been selected, 
the user can view detailed information on metrics belonging to the represented type. 

Video
===============================================================================
![An example of Navigation and Interaction with FindBugs City representation](Code2City.mp4 width="100%")

<!--Results on videos recorded in the indoor area we chose.

<video  width="330" height="446" style="float:left" controls>
	<source src="monica.mp4" type="video/mp4">
</video>

<video  width="330" height="446" style="float:right" controls>
	<source src="nicola.mp4" type="video/mp4">
</video>

<p style = "margin-top:500px;">Results on videos recorded in the outdoor area we chose.</p>

<video  width="330" height="446" style="float:left" controls>
	<source src="Iphone1.mp4" type="video/mp4">
</video>

<video  width="330" height="446" style="float:right" controls>
	<source src="Iphone3.mp4" type="video/mp4">
</video>-->


Resources
===============================================================================
<!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="markdeep.min.js"></script><script src="https://casual-effects.com/markdeep/latest/markdeep.min.js?"></script><script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>
<style>
	
	.md div.imagecaption {
		text-align : center; !important
	}
	
	.shortTOC {
		display : none;
	}
	
	.foo table{
		width: 100%; 
	}
	
</style>

<div class="foo">
| Download |                                                                  Description                                                                  |
|:--------:|:---------------------------------------------------------------------------------------------------------------------------------------------:|
|<a href="Code2City.zip">Code2City</a>    					| Windows Binary Files (No VR mode) |
|<a href="online_material.zip">Materials and Raw Data</a>	| Experimental Materials and Raw Data |
</div>

Bibliography
===============================================================================
[#Capece2017]: Capece, N., Erra, U., Romano, S., & Scanniello, G. (2017, June). Visualising a software system as a city through virtual reality. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics (pp. 319-327). Springer, Cham.

[#Wettel2007]: Wettel, R., & Lanza, M. (2007, June). Visualizing software systems as cities. In Visualizing Software for Understanding and Analysis, 2007. VISSOFT 2007. 4th IEEE International Workshop on (pp. 92-99). IEEE.

[#Stoakley1995]: Stoakley, R., Conway, M.J., Pausch, R.: Virtual reality on a wim: Interactive worlds in miniature. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 265-272. CHI '95, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1995)

[#Wettel2008]: Wettel, R., Lanza, M.: CodeCity: 3D Visualization of Large-scale Software. In Companion of the 30th International Conference on Software Engineering. pp. 921-922. ICSE Companion '08, ACM, New York, NY, USA (2008)