Szilveszter Szeleczki

Directions in the Development of Virtual Reality
and Its Military Applicability

Szilveszter Szeleczki

Directions in the Development of Virtual Reality and Its Military Applicability

Budapest, 2023

This work is funded by the National Fund for Research Development and Innovation through
the Thematic Programme of Excellence 2020, project number TKP2020-NKA-09.

Acknowledgment: This document prepared with the professional support
of the Doctoral Student Scholarship Program of the Co-operative Doctoral Program
of the Ministry of Innovation and Technology financed from the National Research, Development and Innovation Fund.

Author

Szilveszter Szeleczki

Consultant

János Pölöskei

Published by the University of Public Service

Ludovika University Press

Responsible for publishing: Gergely Deli, Rector

Address: HU-1083 Budapest, Ludovika tér 2.

Contact: kiadvanyok@uni-nke.hu

Managing editor: Katalin Pordány

Copy editor: Zsuzsánna Gergely

Layout editor: Angéla Fehér

ISBN 978-963-531-880-3 (PDF)

ISBN 978-963-531-881-0 (ePub)

© Author, 2023

© University of Public Service, 2023

All rights reserved.

Contents

Introduction

Historical overview of virtual reality

Virtual reality in our everyday lives

Technology background

Main technologies related to virtual reality

Economic perspective

Developer perspective

Social perspective

Possible future vision, the world of metaverse

Approach to virtual reality for military purposes

Foreword

Major development-related military capabilities

Exploring virtual development opportunities

Summary

Conclusions

References

Introduction

Our modern society is undergoing a digital revolution. Digital areas are constantly evolving to support people’s everyday lives. Revolutionary changes in the technologies becoming available to ordinary people have filtered through to everyday life, from digital payment methods to the virtual world of everyday transactions. Information is virtually present everywhere in people’s environment. We see information in our natural environment, for example about animals, roads, other people, as well as in artificially created environments such as our digital devices (be it a phone, a tablet, or a desktop computer). So information is virtually everywhere, en masse.

There are many technologies available today to perform information-related activities. The collection, storage, processing, modification, refinement and transmission of information can nowadays be supported by a variety of devices and their networked systems. It cannot be argued that without these infocommunication networks, our society would have great difficulty functioning. It is no coincidence that our current information-centric society is called information society. [1] In our information society, the development and modernisation of infocommunication technologies is significant and particularly intensive; moreover, the related needs and requirements are constantly changing. In the context of information protection, the term cyberspace and related concepts are widely and well known, all of which are the result of the expansion and development of the information space. In cyberspace, there is a multitude of operational possibilities that arise from the interconnection and interaction of the real and the artificially created virtual spaces.

Nowadays, virtual spaces pervade our society and have a significant impact on our society in many ways. As it is true in real space, so it is in virtual spaces, sharing and living experiences have become a natural need in everyday life. In order to be part of a virtual community, it is required that people create a virtual entity of their own personality. Among the related technological developments, the immersive technologies stand out. In the sense of immersive technologies, virtual reality, augmented reality, mixed reality and extended reality are to be discussed. All four areas are quite popular, especially virtual reality and augmented reality. Immersive technologies are of particular interest to people because they can provide experiences that real space cannot or can only partially provide to them. It is important to note that virtual environments already have and will continue to have a significant impact on people’s everyday lives, and it is therefore useful to consider and interpret the technologies involved in several ways. Such multiple approaches will of course help to create a comprehensive picture of the features and capabilities of each of the technologies mentioned, thus helping the people to appropriately balance the use of these virtual spaces and the real space at the same time. In order to achieve this goal, this document will also take a number of so-called perspectives to address what is considered to be the more relevant knowledge of these technologies in question. The versatile and detailed study of immersive technologies is clearly beneficial, independently whether you are a single user of them, a developer of environments based on such technologies, or otherwise interested in this field.

Virtual reality is extremely popular and it is natural that users do not pay much attention to the deeper level of understanding of the technology beyond the experience. Following this line of thought, the next chapters will introduce virtual reality and related immersive technologies from different perspectives, with a variety of concepts and interpretations. After presenting the historical background of virtual reality, the main technological basics, the related technologies and the possible future direction will be discussed.

The economic, development and social perspectives have been selected for the study of immersive technologies. Currently, there are few documents in which one can read about the multiple perspectives on virtual reality in one place, even though the technology (virtual space) has many implications for society. In addition to the understanding of immersive technologies, it is important to note that a number of potential societal impacts can and should be considered in their application. As these are products that represent innovation, it is only natural to look at various statistical indicators and to assess their economic potential. The economic approach also involves looking at the products that are considered to be important, which is why three products from different well-known companies have been selected to provide the reader with an insight into their basic characteristics. From a developer’s point of view, a deeper understanding of virtual and augmented reality as two of the most popular technologies and the design and development testing processes and methodologies required to develop the immersive technology itself are specifically discussed. As with all development, the existence of an appropriate development environment is essential for these technologies. In addition to this, social technology assessment is also important because it is worthwhile to design and implement features that are beneficial to this area. In addition, it is also important to take a social approach because, although often unspoken, it is essential to be aware of the feelings that the human brain has about the world of reality (the natural environment). Personal behaviours in virtual environments today clearly shape people’s mental states and personalities, resulting in similar behaviours in real environments. By creating feelings, colours, shapes, sounds, virtual reality helps the human brain to discover previously hidden connections in complex data sets. Users will be able to collect, store, process and transmit information faster and more flexibly. Pervasive technologies can be seen as a milestone for information services, as they enable the artificial extension and augmentation of the real space information environment and systems in our everyday lives. There is a growing interest worldwide in the potential applications of this technology, of which several examples are presented in this paper.

In addition to the civilian applications, it is important to highlight the potential military applications of these technologies, which are dealt with separately in this document. Military operations are also surrounded by a vast amount of information and, of course, a very large number of information communication tools and systems. Military developments are clearly intense in our society today, particularly in the quest for continuous modernisation and innovation. The second half of this study therefore focuses on potential military applications in the context of the use of immersive technologies. After a brief introduction, related development areas of specific interest to military capabilities are examined, as well as different military sectors and units with specific ideas to share with the reader.

By studying virtual reality and with it, immersive technologies for civilian and military purposes, the aim is clearly to provide the reader with a comprehensive picture. As a final result, what follows is a great starting point for those interested in the field of immersive technologies, in terms of multiple approaches. Therefore, as described below, the document can be taken both as a guide with conclusions and as a technology overview.

[1] “An information society is a society where the creation, distribution, use, integration and manipulation of information is a significant economic, political, and cultural activity. The aim of the information society is to gain competitive advantage internationally, through using information technology in a creative and productive way” (Definitions.net: Information Society. s. a.).

Historical overview of virtual reality

The historical overview of virtual reality can be linked to the need and process of virtualisation; therefore, it is worth saying a few words about the emergence of the needs and conceptual interpretation of the field. It is a technology that is now quite popular and its everyday use has become natural. But let us stop here for a question: What does virtualisation actually mean? It seems natural, yet the more people you ask, the more answers you get. Virtualisation is defined as follows: “Virtualization is the creation of a virtual – rather than actual – version of something, such as an operating system (OS), a server, a storage device or network resources.” [2] Virtualisation is an activity that does not take place in physical space, it is not a tangible process. Many people think of it as an extension of physical space, an approach that comes quite close to lumping the needs of virtualisation together in this thinking. In order to make virtualisation as a whole easier to understand (from its inception to the emergence of today’s modern technologies), it is useful to look at the main aspects of virtualisation, starting with the needs, which can be divided into two main areas:

–the need for virtualisation of computing resources

–the need to create virtual reality, virtual space

With the development of electronic components, computers with ever-increasing computing power have come onto the market. Components became smaller and smaller and at the same time, more and more capable. The evolution of data collection, storage, processing and transmission functions has led to an increasing variety of needs. Pushing the limits of computers’ capabilities, there is a need for more efficient use of resources, which is now a well-known virtual solution. To make efficient use of computing power, seven main components can be virtualised: desktop, server, storage, network, application, hardware and the operation system.

Desktop virtualisation allows multiple virtual machines to run desktops on the same physical and virtual server. Server virtualisation allows management in a cloud. It creates a virtual server in a physical computer. Storage virtualisation reducing costs associated with space in a centre. The network virtualisation uses physical and virtual components at the same time to create a hybrid network and make efficient administration of the network (through software switches). The application virtualisation helps to create a virtual instance of the applications (needed for business) to keep app software off from the local operating system. Finally, the hardware virtualisation aims to make and run different operating systems (creation of virtual machines) on the same hardware. It allows to use the processor simultaneously by more than one user. In addition to all this, a special type of virtualisation is the operation system virtualisation, that can run multiple operation systems instead of having a dedicated server for each system. Virtual environments and virtual spaces are becoming more and more natural for the efficient use of resources, especially for operational tasks. The efficiency of virtually shared resources has certainly contributed to the evolution of the way of thinking about virtual space. The development of virtual services has been accompanied, so to speak, by an examination of the dimensional potential of virtual space. Clarifying the functional needs of virtual space and exploring possible platforms for it are essential measures for the development of the technology. Virtual space, and with it virtual reality, is currently considered a cutting-edge technology, but the history of its development goes back much further than many people realise. It is important to note that the two main trends in the development of virtual solutions is to make computing resources more efficient, and the development of virtual reality are parallel to each other. In the following, detailed main historical events of virtual reality are described.

The first related device was certainly the stereoscopic dissipative (also called mirror stereoscope), invented by Sir Charles Wheatstone in 1838, which can be considered the predecessor of head mounted displays. The invention as you can see in Figure 1 allowed users to see a separate image in each eye (see E and F marks on the picture) creating a wider, three-dimensional image.

Many years later, in the early 1900s, the advent of aeroplanes would see further claims close to the technology. The ancestor of educational flight simulators was made a reality by Edwin Link. The simulator provided cockpit exposure and pneumatic pumps that gave an artificially created sense of the flight environment, a realistic experience for pilots. Training systems can support skills development, practice and maintenance in an efficient way (location, time, equipment, running costs) by complementing the traditional training tools.

Figure 1: The stereoscopic dissipative

Source: Duncan (2019): op. cit.

This has proved to be beneficial, as pilots have been able to develop their skills without actually using an aircraft. It is therefore important to stress that the military application of this technology has also begun to be intensively applied. “In 1929 Edward Link created the “Link trainer” (patented 1931) probably the first example of a commercial flight simulator, which was entirely electromechanical. It was controlled by motors that linked to the rudder and steering column to modify the pitch and roll. A small motor-driven device mimicked turbulence and disturbances. Such was the need for safer ways to train pilots that the US military bought six of these devices for $3500.” [3] As a point of interest, in 1930 Stanley Grauman Weinbaum presented a science fiction story linked to technology, Pygmalion’s Spectacles. In the story, the author imagined a pair of spectacles that would allow virtual experiences of sight, touch, smell and taste. It is important to emphasise, after the initial focus on vision as a perception, the focus has increasingly shifted to the study of the other human senses. The initial testing of the senses in an artificial environment has led to further technological developments.

The 1950s was a very important period for the capabilities (creative ideas and initial implementations) of immersive technologies as we know them today. One example in civilian applications is the theatre cabinet Sensorama, which was designed by Morton Heilig. The aim was to immerse the user completely in the film, using stereo speakers, a stereoscopic 3D display, odour generators, fans and a vibration chair, among other things. The Sensorama in Figure 2 was called the cinema of the future. The product was accompanied by short films such as Belly Dancer, Motorcycle, Dune Buggy.

Figure 2: The theatre cabinet, Sensorama

Source: Jeremy Norman’s HistoryofInformation.com (s. a.): op. cit.

In the 1960s, the development of technology began to intensify, based on earlier ideas. The real breakthrough for virtual reality products was Morton Heilig’s patented invention of the Telesphere Mask in 1960, the world’s first head-mounted display. The invention had three-dimensional, widescreen images and stereo sound (stereoscopic technology), which gave rise to many new ideas for many people interested in technology. It is important to note here that virtual reality and augmented reality, two popular development trends today, are presumably based on the later idea of both the abovementioned, and have started to develop for both civil and military applications. “The history of VR has often been influenced by the defense industry. One of the first VR projects was developed in the 1960s for a US military combat system. Virtual reality has always played a significant role in the military and was adopted by all services: army, navy, and air force. It is mostly used for army training purposes, but there are other use cases where VR is a powerful tool.” [4] At this time, the analysis of user experience needs and the mapping of related technological capabilities were already going on in parallel and intensively.

Among the developments of the 1960s, it is important to highlight the appearance of the first motion tracking system, which was implemented in 1961 as the “Headsight” product developed by the Philco Corporation. The product was not developed to create virtual reality, but it allowed for getting realistic experience, particularly military operations. The two main components of the product were a video screen for both eyes and a magnetic motion tracking system linked to a camera. Later, in 1965 computer scientist Ivan Sutherland presented his vision of the Ultimate Display. The concept aims to reproduce reality so that the user cannot distinguish between the actual reality and the virtual reality created. It is important to note that this interaction between man and machine has become a key factor in the development of technology. Around the world, documentation written at this time is seen as the basic blueprint for virtual reality. In 1969, Myron Krueger (considered the computer artist of virtual reality) described his experiences with projects such as Glowflow, Metaplay and Psychic Space, Videoplace, which were major advances, particularly in the development of the so-called “Videoplace” technology.

In the 1970s, aircraft simulators continued to evolve. In 1972 General Electric started to produce a computer flight simulator. The Videoplace project shown in Figure 3 led by Myron Krueger, opened up further and quite new possibilities for human–machine interaction. Research during this period can be seen as an artistic exploration of visual experiences. As you can read about the project: “Videoplace consists of two rooms that could be in the same building or on the other side of the planet. When a participant enters they immediately see themselves projected on a screen in front of them, as well as the projection of anyone in the other room. Both of the participants see the same image. The participant can move their image around on the screen by moving themselves and can interact with the other participants image as well. Either participants image could be resized, rotated, have the color changed, and they could also interact with objects that were completely virtual.” [5]

Figure 3: The Videoplace project

Source: Lee–Lee (2014): op. cit. 433.

In 1977, a program was created that allowed users to roam around a city, Aspen (USA, Colorado), by taking photographs of the city. The program was called Aspen Movie Map, which is similar to the now well-known Google Street View. It was the first human–machine interactivity that made it clear that virtual reality could take users to other places. Meanwhile, in terms of military applications, in 1979, the McDonnell-Douglas Corporation integrated virtual reality for military use into a head-mounted display, called the “Vital” helmet. The head tracker of the device already tracked the pilot’s eye movements, interacting with computer-generated images.

In the 1980s, the production and sale of head-mounted displays and associated gloves with sensors, among other things, became a prominent feature. In 1985, Jaron Lanier and Thomas Zimmerman founded VPL Research, the first company to deal specifically with these products from a financial, business perspective. Concerning Jaron Lanier, it is worth noting that he both coined and popularised the term “virtual reality”. The gloves were linked to a computer system and used optical sensors to detect finger movements. These gloves were the forerunners of the so-called “data gloves” today. The previously invented finger-tracking glove “Sayre” (the first wired data glove), shown in Figure 4, was developed by Richard Sayre, Daniel Sandin and Thomas DeFanti.

Figure 4: The Sayre finger-tracking glove

Source: Electronic Visualization Laboratory (1977): op. cit.

From a military perspective, a flight simulator called “Super Cockpit” was developed between 1986 and 1989, which used computer-generated three-dimensional maps in the cockpit, as well as infrared and radar images. The helmet had a tracking system with sensors that allowed the pilot to control the aircraft by speech, gestures and even eye movements. The pilot could see and hear everything in real time. In 1987, British Aerospace developed a virtual cockpit with speech recognition. It is important to point out that in 1989 NASA also took notice of virtual reality technology and created the so-called “View” project. As Figure 5 shows, the project aims to use the product for astronaut training. “The Virtual Interface Environment Workstation (VIEW) is a head-mounted stereoscopic display system in which the display may be an artificial computer-generated environment or a real environment relayed from remote video cameras. Operator can ‘step into’ this environment and interact with it. The DataGlove has a series of fiber optic cables and sensors that detect any movement of the user’s fingers and transmit the information to a host computer; a computer generated image of the hand will move exactly as the operator is moving his gloved hand. With appropriate software, the operator can use the glove to interact with the computer scene by grasping an object. The DataSuit is a sensor equipped full body garment that greatly increases the sphere of performance for virtual reality simulations by reporting to the computer the motions, bends, gestures and spatial orientation of the wearer. VPL [6] created its own version, the eye display in NASA’s helmet system, the EyePhone, a head-mounted stereo display.” [7]

Figure 5: NASA’s VIEW project

Source: Rosson (1990): op. cit.

Developments in the 1990s were also significant, as in the past. Products have increasingly become available for public viewing and purchase. It should be noted that Jonathan Waldern exhibited Virtuality (a line of virtual reality gaming machines, that were produced by Virtuality Group), a virtual reality gaming machine, in London. So virtual reality was present in video arcade games. These were the first arcade machines to feature virtual reality technology, allowing players to play in a three-dimensional gaming world, and the first mass-produced virtual reality entertainment system. The machines were already networkable for multiplayer games (for example a popular game was Pac-Man). In 1991, the work of Antonio Medina (then a NASA engineer) led to the development of a virtual reality system that allowed a Mars rover to be driven (the time delay was taken into account). This solution was called “Computer Simulated Teleoperation”. It was also during this period that several companies in the entertainment industry started to focus intensively on technology. The popular SEGA company introduced its own virtual reality headset in 1993, developed for the SEGA Genesis console. Unfortunately, technical development difficulties meant that the device remained forever in the prototype phase (although four games were developed for the product). Later, in 1994, the company released the SEGA VR-1, a motion simulator arcade machine. In 1995, the Nintendo company also got involved with the technology, launching the Virtual Boy console, a 3D monochrome video game player. The Virtual Boy shown in Figure 6 was the first portable device to display 3D graphics.

Figure 6: Virtual Boy, the pioneer of the 3D display

Source: TheGeek (2020): op. cit.

Unfortunately, the product’s development was halted after a year due to the lack of colour graphics and software support, as well as the inconvenience of using it. It is important to note that in the meantime, virtual reality headsets for using at home have already appeared, for example Virtual IO has released its I-Glasses product shown in Figure 7.

Figure 7: Virtual IO I-Glass

Source: V-Rtifacts (1995): op. cit.

A particular use of the technology has also attracted attention, with the virtual reality product being promoted as a treatment for trauma. In 1997, Georgia Tech and Emory University used the technology to help war veterans treat PTSD (post-traumatic stress disorder) by controlling the factors that trigger trauma. Virtual reality has become useful for therapists to know what patients are seeing and experiencing. It was called Virtual Vietnam at the time. “Virtual reality exposure therapy (VRET) is patterned after real-world exposure therapy techniques used to treat common phobias such as fear of heights. By creating a virtual environment where the level of stimuli (such as helicopter flights and gunfire sound) can be carefully controlled by a trained therapist, VRET allows clinicians to apply exposure therapy techniques in a virtual setting.” [8] It is important to mention the virtual reality-related film The Matrix, released in 1999. The characters in the film live in a fully simulated world, which is difficult to separate from the real world (a key motif). As Figure 8 symbolically illustrates, The Matrix has a significant cultural impact on virtual reality, which has encouraged further thinking about the technology.

Figure 8: The protagonist (Neo) is in virtual space, in the matrix

Source: Van Meeuwen (2012): op. cit.

In the 2000s, further, already noticeably intensive development took place. An important event to be mentioned is the development in 2001 of the SAS Cube, the first computer-controlled cube room, based on the ideas of Maurice Benayoun and David Nahon. The aim was to enable the user to visualise the artificially generated virtual environment in the cube, in all spatial directions. A few years later, in 2007, the well-known and famous Google LLC also introduced the technology. The company expanded its existing two-dimensional map service with 360-degree street images (Street View). The pictures were taken by cars specially upgraded for this purpose (which has since been continued worldwide), using a unique camera system. In 2010, Palmer Luckey (American entrepreneur; founder), then just 18 years old, led the development of the first prototype of the Oculus Rift headset, which offered a 90-degree field of view. The product relied on the specific processing power of the computer to produce the images. In the same year, Google announced that the Street View service would also include a 3D mode, which was being developed for the Oculus Rift headset. In 2012, Palmer Luckey launched a Kickstarter campaign to develop the Oculus Rift, raising nearly $2.5 million. It is important to note that from this moment on, the demand for the development of virtual reality products (despite initial failures, see SEGA) has skyrocketed, a virtual reality revolution has been started. Facebook, founded in 2004 under Mark Zuckerberg, acquired the aforementioned Oculus in 2014. As a result, the social media and virtual reality fields began to converge for development.

Meanwhile, Sony announced that it had started to develop a virtual reality add-on for its very popular PlayStation 4 console. By 2015, the technological opportunities had become widely popular with potential users. For example, The Wall Street Journal launched a virtual reality-based rollercoaster that followed the ups and downs. “This year was full of achievements for The Wall Street Journal’s interactive graphics team. After winning the Pulitzer Prize as part of a large team that investigated Medicare billing practices, we continued experimenting with new tools and technologies, including virtual reality and 3D data visualization.” [9] 2016 is a rather important year in the history of virtual reality, as the so-called haptic sensing was previously underdeveloped. The sensor-based tracking allowed users to move freely in a space first implemented by the HTC VIVE SteamVR headset, shown in Figure 9. Haptic sensing involves thinking about the possibilities for human–machine interfaces. These interfaces allow users to interact with the computer through touch and gestures.