Computer-mediated technology attached to humans in order to enhance and empower humans

A new technology called “human augmentation” is attracting attention. Rather than simply using hi-tech machinery to assist with activities and daily life, it will need to give humans the feeling that they themselves are actually operating a machine or device, while also providing the sensation that their actions are influencing their surroundings. The key point is computer-mediated interaction between one human and another, or between humans and society. Scientists say that, if humans’ ability to operate machinery based on this approach increases, it might one day be possible to engage in completely different endeavors than has been the case until now, even in the fields of health and nursing care.

The term “human augmentation” refers to technology systems that enhance and empower humans’ mental and physical functions through the use of technologies attached to humans, including IT, sensors, and robots. The terms “human augmentation” and “augmented human” first began to be used in 2010 or thereabouts, in the field of human computer interaction (HCI), which is the research of interactions and interfaces between humans and computers.

Human augmentation technology is a collection of diverse technologies

In the realm of human augmentation, computers mediate in some form between humans and society, humans and the environment, or one human and another. No computers are involved when talking face to face, but computers are involved as a third party if we use a smartphone to make a phone call. 2010 could be said to be the point when the stage was set for the emergence of human augmentation, as that was the year when technology of this kind spread worldwide.

However, the definition of the term was still unclear at that time, and individual researchers developed a diverse array of technologies (Figure 1). We were researching “superhuman sports,” which involve the use of powered clothing —— suits of artificial muscles —— and wearable robots to augment physical capabilities. Other researchers focused on “tele-existence” technology, which allows the user to effectively “possess” a robot-like entity in a remote location and cause it to act just like the user. Augmented reality (AR) is a technology often used in games that superimposes virtual images on the real world. AR, too, is a type of human augmentation technology. Another form of technology that emerged was brain-machine interface (BMI), in which brain waves and neurons are detected, then analyzed using a computer to decode intentions and perceptions; conversely, signals from the computer may be used to cause movement in the human subject.

Figure 1. Examples of human augmentation technologyHuman functions are divided into the four categories of physical functions, sensation, cognition, and communication. The diagram shows technologies that bolster and supplement these functions, along with situations in which they are actually used. The concept of human augmentation consists of various technologies that complement each other.

As development of the individual component technologies of human augmentation continued to progress thereafter, human augmentation itself expanded as a research field. As such, the Human Augmentation Research Center was established within AIST in 2018 and I was appointed as its director.

This realm brings together a diverse array of technologies, which we define collectively as a network of technologies to be attached to humans, in order to enhance and empower humans. The individual technologies are classified into four human functions: physical functions, sensation, cognition, and communication. I believe it was probably in order to combine and leverage these technologies to contribute to society that the need to define the technological field of human augmentation arose.

Allow me to explain the aforementioned definition “technologies to be attached to humans, in order to enhance and empower humans” a little more. What are the core concepts and approaches of the tele-existence, BMI, and wearable robotics technologies referred to as human augmentation? While it is not the case that all researchers think the same way about these, I myself believe that the two key elements are sense of agency and sense of self-efficacy.

Sense of agency and sense of self-efficacy

Sense of agency refers to a sense of unity between human and machine. It means being able to feel as though a robot in a remote location, or a robot that you wear, or a BMI or similar device is part of your own body. If you wanted to fly through the sky and used Doraemon’s Takecopter or jet propulsion with the aid of a rucksack-like jet pack, that would be human augmentation. On the other hand, it would be difficult to describe an airplane as human augmentation. The difference is the agent of action. In a plane, which allows us to sit back and relax as the pilot conveys us to our destination, the sensation is that we are being transported, rather than that we ourselves are flying through the sky. If we followed this benchmark, driving a car would be human augmentation, but self-driving could not be described as human augmentation.

The other key element is a sense of self-efficacy. This means being able to feel that what has happened there is the result of your own actions. It is a concept similar to a sense of agency, but to take an example, if I have used a text-generative AI to write something, I cannot feel as though I myself have actually written it. However, if a generative AI transcribed the text of one of my lectures, these would undoubtedly be my own words and I would be able to feel as though I was reaching out to and influencing those who read them. This is a sense of self-efficacy.

Whether in the case of the aforementioned tele-existence and wearable robotics, or in the case of BMI, these two elements together mean being able to feel as though the device is part of you, and also that the results of influencing others or the world via this technology are your own work. In terms of perceptions on the part of humans, the concept of human augmentation needs to include a sense of being the agent of action and a sense of unity between human and machine, rather than being controlled by a computer or other device.

I reached this conclusion as a result of an experiment I conducted when I first began my research into human augmentation (Figure 2). This experiment involved kicking a ball toward a goalpost while wearing artificial muscles, in a sport called “Superhuman Penalty Kick.” Even if a person’s strength is increased by artificial muscles, it is only about 1.2 times their actual strength. However, we added a device that, the moment the player kicked the ball into the goal, made a loud noise —— sort of a boing! —— of a kind you would not get in real life, as though the ball had gone in at a tremendous speed. Once we did that, the researchers who had tried out the experiment wanted to try it again and began to line up in front of the device. This device that uses a component technology of human augmentation provides a sense of its possibilities for bolstering human strength, making people more active, and increasing their motivation to continue engaging in such actions.

Figure 2. Mechanism for increasing motivationThe device not only boosts muscle strength, but also creates an unexpected feeling of elation due to the sound effect of the ball causing the goalposts to rock.

We also discovered a new attraction of human augmentation when we held an event for local citizens at the Center. It works like this: in a scenario involving a battle between an action hero and a monster attacking the earth, participants get involved in the fight and borrow human augmentation technology to face off against the monster.

Avatars influence actions in real space

Scenes of the hero fighting and the monster’s attacks are shown on a monitor. The participant tries pulling on a rope attached to the monster’s tail in an attempt to stop its attack. However, it is too strong for them. Next, they put on artificial muscles and try again to pull its tail. While they exert more strength than they did the first time, it still is not enough. For the third attempt, they put on glasses of the same design as those worn by the hero. When they do so, an image of the hero after a transformation is shown on the monitor and the hero’s movements are synchronized with those of the participant, as though the hero was part of them. The device is equipped with motion capture technology, which perceives the participant’s movements. When they put it on, we discovered something astonishing: if the strength of their first attempt was measured at, say, 10, their second try was around 12, but their third attempt was far stronger than the first two, at 18 or 19. In other words, the avatar (the representation of the user in virtual space) affected the user’s actions in real space. The technical term for this is the Proteus effect, and one might say that a change in the participant’s feelings caused them to exert more strength than envisaged. While this phenomenon was observed in most participants, it does not occur in the laboratory. It was purely a phenomenon that occurred at the event, for which we had worked out a story. This subsequently became a major point of focus for us when considering human augmentation.

* Motion capture: A technology that measures movement in three dimensions and expresses the position following the movement in numerical terms.

One can imagine that some kind of substance was produced in the brain when the participant transformed into the hero, but the highly interesting thing about this phenomenon when considering the realm of human augmentation is that not only wearing some kind of device, but also introducing an avatar could be an effective means of producing a sense of unity between human and machine, along with a sense of self-efficacy.

One trend in augmentation technology is to focus on humans as a species and look at enabling our flightless species to fly. Further, enabling people with mobility problems to walk, supplementing functions lost due to some kind of disease, and bolstering abilities that have diminished with age also all fall within the realm of human augmentation.

At our center, we also believe that ongoing use of devices that reinforce capabilities might actually enhance or restore a person’s own abilities, so that they could aim to achieve change even without the device.

We are working on a number of social experiments based on this understanding of human augmentation.

One is a mechanism to support nursing care.

We know that, when people who require nursing care continuously use care equipment in daily life, only a small percentage see a deterioration in their levels of care required, whereas the percentage who experience an improvement increases (Figure 3). It is vital to ensure that people can continue to exercise in the course of activities of daily living with the aid of appropriate care equipment. We developed a robotic walker for this purpose. We attached sensors to an ordinary walker, along with a device that would exert force to pull the user up slopes and apply the brakes on downward slopes, so that they would not fall over. With this, we were able to get subjects to walk more.

Figure 3. Usefulness of care equipment for people who require nursing careCompared with those who require nursing care, but do not use care equipment, the percentage of those who have used such equipment whose care needs deteriorate is significantly lower. This suggests the possibility that an improvement in the condition of those who require nursing care could be achieved with the aid of a device that encourages everyone to use such equipment.(Analysis of nationwide nursing care insurance receipt data over eight years from 2006)

Furthermore, we collected data while the walkers were in use and are considering how they might be used in rehabilitation, by varying the degree of support they provide according to improvements in the user’s abilities —— reducing the assistive power of the walker if the distance the user can walk increases, for example.

We are also developing a number of mechanisms to help people take regular exercise to maintain their health.

According to data from the Ministry of Health, Labour and Welfare (MHLW), only around 30% of people exercise regularly. Although there are apps and new tools to encourage people to take moderate exercise, that figure remains stuck at 30%. Consequently, we can see that even though 70% of people know exercise is good for one’s health and that obesity leads to lifestyle diseases, that knowledge does not translate into a mindset that drives them to start taking exercise and continue exercising regularly.

Looking at the psychological profiles of the people in that 70%, they can be classified as either Low Pragmatic (Passive Followers), High Pragmatic (Conformist Followers) or Apathetic (Figure 4). While it is not easy to ensure that the message reaches the Apathetic group, the Low Pragmatic group is the group of people who will, for example, participate if invited, but lack the motivation to continue exercising, which means that it does not become an established habit. The High Pragmatic group, meanwhile, is the group of people who, even though they are motivated to try out a form of exercise because everyone else is doing it, will easily get out of the habit, due to their lack of clear motivation. In both cases, these groups will try participating in exercise a number of times, but gradually stop taking part, for no particular reason.

Figure 4. The psychology of inability to continue with exerciseAlthough people know that exercise is good for one’s health, only around 30% of people have established exercise habits. A different psychological approach is required for other people.

Mechanisms for intensifying feelings of empathy

So, why did people participate in something about which they were not enthusiastic? It might have been because they wanted to show how much they valued their relationship with someone in their social circle, such as a friend, for example. Accordingly, we wondered what would happen if we created a mechanism that intensified feelings of empathy with one’s peers.

Findings in the field of psychology have shown that as people’s sense of identification with others increases, their facial expressions such as smiles become synchronized with them. If the degree of synchronization increases, it could boost empathy and strengthen motivation to exercise. With this as our starting point, we decided to photograph participants’ expressions when exercising together, so that we could measure their empathy levels. We told the instructor about our observations and conducted an experiment in which the instructor induced the synchronization of expressions, from which we found that the Low Pragmatic group’s rate of continued participation actually increased.

Awareness of likes and dislikes can impede improvement, but this, too, is an area where human augmentation technology is useful.

Professor Masahiko Inami of the University of Tokyo conducted an experiment in which participants wore a virtual reality headset and practiced kendama (a traditional Japanese wooden skill toy consisting of a handle to which a wooden ball is attached with a piece of string) skills in virtual space. In a virtual environment in which time alone is slowed down, the ball falls more slowly than in reality after being tossed into the air. At the start, the test subjects had no kendama skills at all. However, as the ball fell slowly in the virtual space, they had time to position the handle for its landing and achieved repeated success. The passage of time in virtual space was gradually increased to a pace closer to real time, and once the subjects were eventually able to achieve success in real time, they then tried out a kendama in real space. Upon doing so, it turned out that in less than an hour, the subjects had mastered kendama techniques. It is also highly interesting that, although a program that corrected the way subjects moved their bodies was not incorporated into the experiment, simply having repeated experiences of success in virtual space gradually led the subjects to use their knees more skillfully. I believe this mechanism is another example that shows the usefulness of human augmentation. More specifically, even people to whom improving performance through low-key efforts does not come naturally can potentially acquire skills if they are swiftly provided with experiences of success.

Right now, the novelty of human augmentation technology and devices is capturing people’s attention, but as this technology permeates through society and becomes more widespread, I believe it will be important to collect and analyze a variety of data, combine technologies, and consider them in their totality. This is because humans are not such simple beings that they can be altered by means of a single technology. The next stage in human augmentation might well be preparing latent mechanisms and using them to transform humans.

(Figures courtesy of Masaaki Mochimaru)