Introduction: Changing Realities
“All reality is mixed reality”, Mark Hansen declares in Bodies in Code (2006).[1] Proclaiming that reality is formed via the mixing of bodies with technologies, Hansen explains that perceptual paradigms and everyday practices are configured by technologies’ applications. Adopting Hansen’s technologically-driven phenomenological approach, our research examines the perception of reality and engagements with public space following one of the most influential technological advancements in the past two decades: Augmented Reality (AR). As AR becomes an increasingly dominant technology, it is essential to examine how we experience the world and how we interpret reality and our immediate surroundings following AR’s emergence. Here we define reality as that which most closely comes out of our lives, or spatially embodied, experience. Therefore, we offer to conduct a reality check: How is our perception of spatial reality reconstructed in light of the rise of AR technology, and what kind of reality is formed?
Roland Atzuma explains that AR “allows the user to see the real world, with virtual objects superimposed upon or composited with the real world”.[2] Today, this is usually achieved with mobile media and smartphone cameras in combination with physical markers or GPS technology. In both cases, AR usually requires users to become active participants, as they must navigate and explore spaces to discover new forms of digital public art. Such experience relies mostly on walking, and while it does possess a performative quality, walking in AR also serves a more practical function: it actively launches and exposes the art. The duality between walking the physical space and its virtual outcome ultimately results in a sense of convergence: AR walking tours enable a new form of spatial performativity, which allows us to experience immediate reality as a convergence of digital elements and material space. As a result, our traditional perception of virtuality and reality as a dichotomy, or as a continuum,[3] is reshaped because virtual elements become an integral part of reality itself. Instead of being immersed in a computerized world, AR breaks the barriers and intertwines virtual elements with non-virtual ones; it is through intersecting both the actual and the virtual domains—as well as by using the navigating body as a focal point—that AR can expose our constructed understanding of space as a kind of virtual environment and alert us of what is called by theorist Henri Lefebvre the illusions of “transparency” and “realism”.[4] Throughout this paper, we demonstrate (1) how AR’s full, real-time synchronization between actual and virtual elements calls attention to the cultural construction of both body and landscape, and (2) how AR walks can be a powerful tool in merging timeframes, and historic and personal narratives. As an artistic tool, augmented interventions can reactivate places and new layers of the city, in which the scale of urban experience is in constant expansion and (re)contraction between citizens, time, and data.
Relational Environments: An Expanded Artistic Practice
Augmented reality becomes an effective tool to motivate people to explore the city. From Pokémon Go! to digital heritage and artistic interventions, AR allows us to visualize the overlapping layers of the city’s social, cultural, and material becoming in real time while walking through it. In this context, Claire Bishop’s concepts of “live installations” and “action outsourced performances” are especially relevant:[5] in AR, the art is launched by the users as they navigate space. This form of interaction shapes a participatory system that is based on geolocated dynamics, which expands the understanding of collaboration, artistic authorship, art spectatorship, and the practice of walking. Walking is the means to activate the art, to create collaborations, and to produce experiences. Engaging with AR, walking turns into a hyperARTive action.
Since navigating the actual space activates the virtual domain, AR emphasis a spatial relationality: in the AR environment, elements belonging to one domain gain meaning and recognition in light of their correlation with the other domain. This point is key in understanding the AR convergence: it not only merges two streams of information into one continuous experience, but it is also used to contextualize all elements involved. Virtual elements are contextualized via their relation with the actual domain, and actual elements are re-contextualized via their relation with the virtual elements that are associated with them in real-time. As a result of this back-and-forth interaction, the AR convergent space, as referred to by Horea Avram,[6] is never abstract and never absolute; it is a space that exists via a dialogue of de-contextualization, re-contextualization, and contextualization of elements in relation to one another, through the continuous experience of engaging bodies. Navigating these spaces, participants can no longer differentiate the digital from the physical nor should they care to do so. Engaging with AR, virtuality is not the opposite of reality, nor does it function as a kind of supplementary content to it. Instead, virtual elements equally and actively participate in the dynamic production of lived-reality.
The project NAMAland by Conor McGarrigle (2010) is a good example to demonstrate these theoretical arguments. Here people navigating the streets of Dublin using the NAMAland AR app were introduced with information about real-estate in the ownership of the National Asset Management Agency (NAMA). NAMA was established by the Irish government with the aim of enhancing the stability of the Irish economy through improving its international credit score. However, NAMA exceeded its budgets and the initiative collapsed. Despite generating some significant economic and social consequences, the reports about the NAMA activity were neither published nor analyzed. These reports are publicly accessible, but the data is unlikely to be understood by the masses.[7] Determined to bring this information to light, McGarrigle gathered and analyzed the NAMA reports;[8] then, he digitally geo-located an icon of the Monopoly Man at the same locations as every real-estate property belonging to the NAMA agency. The monopoly man, along with additional information, becomes visible once the GPS system on the mobile phone identifies the exact location of the app’s user. Since many of the NAMA buildings have been deserted, McGarrigle’s project provides the public with an explanation to the shaping of their actual surroundings. By geo-locating virtual content and aligning it with physical locations, we understand how and why the urban space is as such; in a way, what McGarrigle’s app offers is an enhanced, more holistic sense of space and place. Converging the digital with the physical not only simplifies the information and makes it publicly accessible, but more importantly it also elucidates Dubliners’ current and overall living conditions. By relating to participant’s immediate location,NAMAland contributes to and extends the situatedness of its users: it does not only reaffirm their physical location and acknowledges their bodily existence in a certain spot, but it also exposes their otherwise implicit and consequential placement in a larger setting of socio-cultural and historical-political events. Thus, NAMAland ties not only the physical and the digital into one sense of place, but it also intersects our immediate surrounding with a broader context. Through the AR platform, walking the space enables and activates an entire network of political, historical, economical, and other shaping forces. Space thus becomes much more than a physical location; rather, it is a site of convergence of virtual and actual elements.
Such spatial complexity is directly linked to Jason Farman’s understanding that our embodied experience in the age of new media is gained through technologies’ practice of “implacement.”[9] Mobile technologies—including AR—are in fact locative technologies, as they operate to situate subjects directly and explicitly within a broader context by acknowledging and reaffirming their specific and immediate surroundings. In other words, we understand ourselves as being-in-the-world as a result of and in relation to mobile technologies’ output. The here-and-now of the AR convergence contradicts the somewhere-and-sometime offered by virtual environments. Instead of positioning us as no-ones in a non-world, AR technologies aspire to enhance and reaffirm our specific bodily presence in our specific location.[10] Once the virtual and the non-virtual are continuously and effortlessly synchronized, the material and immaterial aspects of body, space, and technology are converged through the interactive experience offered by AR apparatuses.
These dynamics of data has led McGarrigle himself to acknowledge the AR convergence as an “active space.”[11] However, one can also interpret the AR space as active due to its dependency on subjects’ movement. While many scholars acknowledge that different ways of mapping can reconfigure our relationships with our surroundings, AR uses mapping and spatial navigation in order to initiate a perceptual paradigm of convergence.
AR Walks: Reshaping the City
While certain dimension of the city can be explained or revealed through AR, artists have also been aspiring to completely transform the experience of the urban space and the geo-political standings associated with it. This can be clearly seen in the AR audio tour You Are Not Here: Gaza/Tel-Aviv (2007), created by the Israeli designer Mushon Zer-Aviv and the Palestinian journalist Laila El-Haddad. While this is an early example of AR, it provides us with a similar experience: based on a double-sided printed touristic map, this project allows participant to envision the “views” of Gaza while walking the city of Tel-Aviv. Upon arrival at a point of interest, participants dial a given phone number and a system identifies their location and plays the relevant audio track describing Gaza’s views and sites.[12] Thus, participants are called to envision and navigate Gaza, while in fact, it is inaccessible to them. Seeking to expose a new spatial performativity, this project conceptually shrinks space in order to multiply its meanings. You Are Not Here makes participants walk to in two cities simultaneously, thus asking them to question and contemplate their situatedness. [13] As the map blurs against the sunlight and the audio alienates their sight, this project asks participants to re-think their current geographical position and socio-historical context, while simultaneously reaffirming their being in a specific physical place. While AR is still a premature and experimental technology at this point, AR concepts are still used here in order to provide access to the inaccessible, and to invade, combine, and dislocate spaces through the actions of walking and navigating. Alongside the specific historical and political implications this project may hold stands AR’s unique ability to activate space in order to face us with our constructed perception of it.
The project ARLines of the City by Giovanna Casimiro (2018) asks us to reconsider not only the physical borders and mental constructions of material spaces, but also the links between public art and its location. In this initiative street art is combined with augmented reality, relocating graffiti murals from one city to another. The project took place in Austria, Brazil, and United States, relocating murals from São Paulo to multiple places around the globe through AR. ARLines invites locals and passersby to rediscover and reactivate their familiar public arenas, and to create a mixed landscape. This project combines art, technology, and existing architectural structures to challenge the municipal limits of cities, realities, and sensibilities through rechanneling the everyday practice of walking the streets. AR is used here as a tool, allowing both artists and participants to re-conceptualize the city as an interface, as it comes to include and interconnect multiple spaces and social realities.[14]
ARLines of the City – Vienna (photo from private collection)
Street art is an integral part of the city and of urban aesthetics, and it is also a powerful tool to express local narratives and identities.[15] However, ARLines asks us to reconsider our traditional understanding of street art as a location-based practice. Using AR, graffiti works exceed their originating environment, while Casimiro demonstrates that such overlaps enable endless opportunities for meaning making. In this case, AR re-invents graffiti and street art as a mobile and dynamic practice that depends on viewers’ physical presence and navigation; while AR graffiti is still location-based, this is a layered location composed of intersecting spaces and data flows. Here too, walking becomes a powerful act, putting tangible and intangible places in mutual interaction. Through ARLines, the murals cease to be fragments of other places and realities, but rather they become inseparable parts of our renewed spatial experience as a result of the AR mix. Going back to Bishop’s concepts, it becomes clear that our explorations actually brings life, and perhaps even new life, to those artworks.
Lastly, while AR turns the city into an interface, it enables not only the interconnectivity of different actual places such as Gaza/Tel-Aviv or São-Paulo/Vienna, but it also becomes possible to merge memories and digital reconstructions of heritage with existing urban contexts. Memory of Amnesia Augmented Tour was a beta version of an interactive intervention created in 2015 by Giselle Beiguelman and it illustrates a similar concept.[16] In this artistic intervention, the artists investigate the processes that determine urban memory through the prism of forgetting. Beiguelman and her teamfocus on two recurrent issues of the urban history of São Paulo: (1) the removal or demolition of monuments, and (2) the relocations of urban monuments to other locations in the city. Memory of Amnesia Augmented Tour traced and mapped all the monuments that have been (re)moved in the past two decades back onto their original place, allowing participants to reactivate monuments in-situ.[17] In this case, the act of walking retraces urban pasts as navigating the public space enlivens symbols and events lost in time. Moreover, walking the tour is reminiscent to the act of remembering, while it also has the agency of reshaping existing urban experience and memory. The AR walk initiates a confusion between personal, collective, and digital urban memories, thus conflating bodyand technology with spatial and temporal dimensions. As a result, AR walks questions the boundaries between self and landscape. As walking historian Rebecca Solnit argues: “Moving on foot seems to make it easier to move in time, […] and the passage through alandscape echoes or stimulates the passage through a series of thoughts.This creates an odd consonance between internal and external passage, onethat suggests that the mind is also a landscape of sorts and that walkingis one way to traverse it.”[18]
Memory of Amnesia Augmented Tour–São Paulo, Brazil (photo from private collection
Conclusion
While the overlapping of time and space through digital tools is a constant, ongoing process happening in multiple contemporary urban contexts simultaneously, we must also acknowledge that the reactivation of places is a phenomenon that strengthens not only the artistic and urban experiences but also the local narratives. The practice of walking in the case of AR interventions turns sites into artistic surfaces—into canvases—on which multiple spaces and times interfere with each other to establish a unique and ephemeral work of art. While Augmented Reality technologies are being used in multiple contexts, from marketing to medical services, AR’s unique artistic practice is situated at the crossroad of cultural heritage and public art. It is this position that establishes AR art as an effective (and affective) critical apparatus. Walking with AR, we reuse our physical presence to regain abilities of spatial democratization and occupation. Such augmentations allow us to include other narratives and intertwine them with different fragments of histories and places. By making visible layers we cannot otherwise see, AR enables another way of establishing urban identities and its potential histories.
References
[1] Mark B.N. Hansen, Bodies in Code: Interfaces with Digital Media(New York: Routledge, 2006), 1.
[2] Ronald T. Azuma, "A survey of augmented reality," Presence: Teleoperators & Virtual Environments 6, no. 4 (1997): 365.
[3] Paul Milgram, Haruo Takemura, Akira Utsumi, Fumio Kishino, "Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum," (Paper presented at the SPIE [The International Society for Optical Engineering] Proceedings 2351: Telemanipulator and Telepresence Technologies, 1994).
[4] Henri Lefebvre, The Production of Space, trans. Donald Nicholson-Smith (Cambridge, MA: Blackwell, 1991; originally published: 1974).
[5]Claire Bishop, "Delegated Performance: Outsourcing Authenticity." October (2012): 91-112.
[6] Horea Avram,"The Visual Regime of Augmented Reality Art: Space, Body, Technology, and the Real–Virtual Convergence, " Ph,D. Dissertation, McGill University, 2016.
[7] Conor McGarrigle, NAMAland: Augmented reality phone app, 2010, accessed March 1st, 2017. http://www.conormcgarrigle.com/namaland.html.
[8] Conor McGarrigle, “Augmented Interventions: Re-defining Urban Interventions with AR and Open Data,” in Augmented Reality Art: From an Emerging Technology to a Novel Creative Medium, edited byVladimir Geroimenko, 81-95 (Springer, 2014).
[9] Jason Farman, Mobile Interface Theory: Embodied Space and Locative Media(London: Routledge, 2013), 40, 52.
[10] Jay David Bolter et al, "New Media and the Permanent Crisis of Aura," Convergence: The International Journal of Research into New Media Technologies (12, no. 1 2006): 27-8.
[11] McGarrigle, Augmented Interventions,83.
[12] Zer-Aviv, Mushon and El-Haddad, Laila. "You Are Not Here." http://youarenothere.org/.
[13] Farman, 5, 39.
[14] Giselle Beiguelman, Giovanna Casimiro, Nathalia Lavigne, “The Online Counter-collector, the Open Source Heritage and the Museums of the Unfinished”. ISEA 2017: Conference Proceedings, 821-30.
[15] Guilherme Teixeira Wisnik, "Inside the labyrinth: Hélio Oiticica and the challenge of the ‘public’in Brazil," ARS (São Paulo) 15, no. 30 (2017): 95-110.
[16] This was a part of the larger project Memory of Amnesia, which was coordinated by the Brazilian artist Giselle Beiguelman.
[17] For more information about the project, as well as for the multiple reasons of monuments removal, see: http://www.desvirtual.com/mda/
[18] Rebecca Solnit, Wanderlust: A history of walking(Penguin, 2001), 5.
Liron Efrat is a PhD Candidate at The Graduate Department of Artat the University of Toronto, where she researches Augmented Reality art. Liron is also a resident scholar at the McLuhan Centre for Culture and Technology, and a data analyst for the AR and VR project Imaginary Jewish Homelands
Giovanna Casimiro is a curator, a multimedia artist, and a PhD student at the Faculty of Architecture and Urbanism at the University of São Paulo where she researches the City as Museums interface. Her collaborations include ARTIVIVE app, ZOAN studio, OVNI Studio, LILO.Zone, Boston Cyberarts, and Digaaí Association.