Smart Shift

In 1991¹ German art-cinema director Wim Wenders released his future-facing masterpiece of prescience, ‘Until the End of the World’. Within the convoluted, county-hopping plot, a sub-story involved a device that could record, then play back images ‘recorded’ from someone’s head. While initially planned as a mechanism to enable blind people to ‘see’, it became apparent that it could be used to record and playback dreams. At first people saw it as smart, but then they realised just how addictive it could be. One hapless character was lost forever to the screen, watching their deepest thoughts on repeated loop. Around her the world carried on, regardless.

As we create such inordinate quantities of information about ourselves and our environments, it becomes a relatively small step to use such it to enhance what we see and experience. In much the same way as the car dashboard provides direct feedback to our driving, so technology can offer a dashboard for our lives. Even smarter, potentially, is linking such feedback to what we see, hear and otherwise sense, as we go about out daily existence. Just as computer gamers are used to seeing words and stats hovering around the characters they control on the screen, so we can add data to the people and objects in our field of view.

This, ‘augmented’ vision of vision was first documented in 1945, by a US engineer called Vannevar Bush. Just as the war was coming to an end, he wanted to extend some of his wartime learnings in a more peaceable direction. “It is the physicists who have been thrown most violently off stride, who have left academic pursuits for the making of strange destructive gadgets,” he wrote² at the time. “Now, as peace approaches, one asks where they will find objectives worthy of their best.” One of Bush’s ideas was to create a head-mounted camera, “a little larger than a walnut,” that would enable scientists to record images as they worked. He also suggested a way of converting speech to text, and using a stenotype-like device to make other notes. Bush recognised that his advanced ideas would not immediately become reality. “Two centuries ago Leibniz invented a calculating machine which embodied most of the essential features of recent keyboard devices, but it could not then come into use,” he remarked.

Indeed, it would be some 50 years before Dr Mark Spitzer³ founded MicroOptical, a company built on US Defense research[^ DARPA] funding to create a spectacles-mounted display. Designed (inevitably) for military use, the MicroOptical concept was to use a clever combination of lenses and prisms to provide a large amount of information through a very small screen, positioned close to the eye. In reality, it was also an idea before its time — as well as being very expensive, the display resolution was too low and could result in headaches and feelings of nausea. While the company found itself a victim of the post-millennial financial meltdown, it was not to vanish completely: in 2013 Google acquired the MicroOptical patents⁴, shortly after which, its Google X division — director Mark Spitzer — announced the Google Glass project. With Glass, the company was looking to pick up on augmentation and bring it to the mainstream— while it was announced with much aplomb, only to be withdrawn a year later, before coming back again as a corporate platform in 2015. The bottom line, it appeared, was that the concept still wasn’t good enough to be of sufficient general use.

The phrase augmented reality’ (AR) has been used to describe a combination of technologies, work in tandem. First we need the data feed, of course. Then we need a digital video stream of physical objects or scenes, adding information to the stream in real-time, displaying it on a suitable screen. We may have had a wait, but today, all such pieces are now in place. Right now AR can be achieved with a smart phone with a camera and internet access — a video is captured, uploaded and recognised, then additional information is downloaded to add to what is seen on the display.

Examples of AR at work fall into three groups. The first, which we might call ‘symbol-based’ is the simplest, it is perhaps the most fun. The 'symbol' is a pre-defined, fixed image that can be recognised by a program installed on a smart device, which can then add information in real time — for example adding a 3D avatar, or replacing somebody's head with a cartoon image⁵. This model is not dissimilar to using QR codes⁶ — squares of pixels which can be photographed and interpreted to link to online information. Indeed, examples exist⁷ of using QR codes as the basis for symbol-sensitive AR. ‘Object-based’ AR takes things one step further, in that image recognition software can identify specific objects and then construct a virtual world around them. Examples include Metaio's digital lego box⁸ and apps to show how, say, to remove toner cartridges or other products from their packaging. Finally we have ‘context-based’ AR, which captures the entire surroundings and adds information prior to displaying both. Google Sky Map⁹ is a simple, effective example of what can be done; other obvious applications are for travellers and direction finding, picking up specific street features and identifying the nearest pizza outlet, say. In these cases the video feed is supported by GPS information — so the software doesn't have to work out which street one is on from scratch! And other organisations are looking into visualisation¹⁰ of complex data.

While AR has yet to find its killer app, it continues to develop. All of these models are being tested out in various ways, with augmented capabilities finding their way into games, such as incorporating a camera into a model helicopter and turning it into a virtual gunship¹¹. Gravity-related features¹² or face recognition (as incorporated¹³ in Google Android's Ice Cream Sandwich) are being used now. Indeed, technologies don’t have to move with you — they can sense your movement. The company Leap first came to market with a tiny device called the Motion, which uses two infra-red cameras to detect the location, and movement, of hands — while some¹⁴ saw it as an idea before its time, it is now being integrated as a controller directly into laptop computers. Meanwhile, Microsoft’s Kinect controller can detect and interpret whole body movement, both adding to the ‘augmented’ experience. Augmented reality techniques are also being used in sport, for example by German company Prozone, whose technology is being used by many European football teams to analyse team movements during and after a match.

MicroOptical was among those recognising the need for a more immersive experience, indeed, it was its experiments with the MyVu heads-up display (renaming the company to MyVu) that led to it running out of money. The link, to all and sundry however, was obvious, particularly given the growing interest in three-dimensional, virtual reality environments. 3DVR environments started to come to the fore at the turn of the millennium — Second Life was the poster child of this movement, another area that connected with people’s desires to share some aspects of their lives (including their desires) in a virtual environment. Unfortunately, Second Life also appeared to be an idea before its time, reaching a point beyond which it started to have less point. Of course it is typical of the technology industry to try technology combinations and see what sticks[^ Sites like Augmented Planet speculate about integration of near-field communications.], but the lure of the immersive environment refuses to go away. The Oculus Rift is the latest in a series of attempts to deliver on Bush’s original ideas — the brainchild of Palmer Luckey,. In a style characteristic of many techno-entrepreneurs, Luckey left college to found OculusVR in 2012, starting a Kickstarter campaign which raised $2.5m in a remarkably short period: his lucky break came when he met gaming industry hero John Carmack, founder of Doom, following which (one way or another) the project has seen a meteoric rise, culminating in being bought for $2 billion by Facebook. The Rift has been some 4 years in development, during which time a number of other technologies have been advancing — not least display resolution, with today’s mobile phone screens working at scales almost invisible to the human eye. In addition, the Rift now comes with built-in devices that enable the display to ‘move relative to the head position of its wearer — that is, so-called accelerometers.

Accelerometers also have a long heritage. McCollum & Peters created¹⁵ first ‘resistance bridge’ accelerometers in the 1920s, weighing about a pound — their use was perceived for engineering, such as sensing for movement in bridges. It was the advent of silicon-based (“Micro-electromechanical systems” or MEMS) accelerometer technology that changed the sensor world in the 1980s, adding to the pool of sensors that could drive data into the Internet of Things and, of course, providing a basis for a number of unexpected uses of today’s mobile phones — such as the plethora of apps now available to help you with your golf swing. Accelerometers are finding its way into gloves and all kinds of other appendages (it should come as no surprise that the porn industry is investing a great deal of time and money in a variety of ‘attachments’ — another form of ‘augmentation’).

As such ideas illustrate, this more immersive world, combining real-life and computer-generated information, can deliver a great deal more than simply providing heads-up displays. It looks probable that AR will trigger a resurgence of the virtual worlds we saw at the beginning of the millennium. One has to wonder what might have happened if Second Life had become popular at the same time as the rise of the virtual reality headset — indeed, the site is still online. But now, with computers more powerful and with interfaces more engaging, such immersive environments can have a second bite of the virtual cherry. As is also the case with massively multiplayer role playing games (MMORPGs) such as Lord of the Rings Online and World of Warcraft, themselves going through a period of reducing demand. All that people are waiting for, potentially, is appropriate kit.

But where will we go if we get it? For better or worse, we are undoubtedly experiencing a closing of the gap between the real and the virtual world. While it’s worth thinking about the two separately, self-proclaimed¹⁶ ‘social media theorist’ Nathan Jurgenson warns against what he calls ‘digital dualism’ — that is, seeing the data and ourselves as separate. “I fundamentally think this digital dualism is a fallacy. Instead, I want to argue that the digital and physical are increasingly meshed,” he said in 2011. At the time, just a few years ago, his words may have been controversial, but now they appear prophetic. “A new heuristic for human experience now blends physical and virtual space in personal, asynchronous time and physical and virtual space in group oriented, synchronous time,” explain¹⁷ Sally A. Applin and Michael Fischer, anthropologists at the University of Kent.

The potential for such technologies are profound. Surgeons for example can shrink to the size of the cell structures they are looking to repair, or the tumours they are trying to destroy. Examples that aren’t particularly exciting are nonetheless profound: consider, for example, the use of Zebra Technologies’ AR capabilities, which enable warehouse managers, to keep an eye on their shelves of stock with immediate information about inventory levels, wrongly catalogued items and so on. And meanwhile, yes, Astronauts can travel to distant planets, and geologists can descend deep inside the earth without having to be physically present. At the same time however, they — we — will feel that we are present. As many philosophers have said, “what we perceive as real” is more correctly stated as “real is what we perceive”. It appears highly likely that we shall be able to see and experience far beyond our current field of vision.

As both AR and VR find their way more into the mainstream, some downsides will inevitably emerge. For example there are clear privacy questions around using facial recognition: right now we can sit on public transport or indeed, walk into a shop with relative anonymity, but this could easily change if, say, our facial features could be mapped onto a database of images. Cases of mistaken identity are also possible — what if you happened to look very similar to a known stalker? There's also the possibility of the AR equivalent of social networking takes hold: when Facebook acquired^18 the Oculus Rift, there was much consternation the our very movements could be sold to third parties, for example. We always have the option to turn such facilities off, but we can’t stop other people from using then, and it is in providers’ interests to discourage us from doing so. And indeed, we may like our augmented lifestyles too much to want to return to merely existing.

And what about our children, born into this augmented world? Examples of AR books for kids already exist^19, but what about when they are better able to navigate the immersive world than their parents? Panic not, we are not yet ‘there’ — the linkage of data and the physical world has some way to go. However, as AR and VR start to integrate other forms of information and as new applications are brought to the fore, we may yet reach a point where we drop the A or V, and it simply becomes 'reality', at which point the bar of what augmentation means has to rise.

The next couple of years are going to see some significant advances in both AR and VR, as screens become sufficiently readable, mobile data transfers sufficiently fast and local processing (that is, on the devices) sufficiently capable to tie all the necessary pieces together. The ramifications may be most visible in the consumer space but it is in industry — in healthcare, policing, education — that the impact will be most profound. How precisely will it be felt? Perhaps we don’t have to look any further than the many books, games and films that have already been created that cover this topic, from Tron to the Matrix (and of course Wim Wenders’ own magnum opus), Myst to Warcraft, John Brunner’s Shockwave Rider to Tad Williams’ Otherworld . What we learn from such films, in general, is that the fight remains between good and evil, between acts of immense inner strength to overcome situations of utter peril. Perhaps this suggests a failure of our own imagination to fully understand how our behaviours might change; or more likely, the fundamental facets of the human psyche, honed over millennia, will pervade whatever the backdrop, in the physical or digital world.

Whatever the ramifications, AR and VR are coming — but even then, technology isn’t stopping with simple augmentation or the creation of immersive worlds. Not only will we be able to interact with the environment in new ways, we will also be able to create it. Welcome to the world of 3D printing.



• read this again :)