2020 has been the year of social distancing and fist bumps. In a society in which COVID-19 is limiting physical contact, what meaning will the sense of touch come to have in our lives in the future? As research into technologies capable of replicating the sense of touch progresses, shedding light on the mechanisms by which touch is processed in the brain and turned into a physical experience, we will come to realize that a physical experience and haptic impression are rendered indivisible via the mind.

Optical illusions are a well-known fact. Our brain might fill in a gap, causing us to see a line that does not exist, or we might misperceive colors or feel that something is leaning at an angle when it is not. These are just some of the many illusions that have been identified.

However, there are illusions other than optical ones. They can also affect our sense of touch. Recent progress in haptics research is driving the discovery of a series of tactile illusions.

Experiencing tactile illusions

One famous tactile illusion is the velvet hand illusion. Place your hands on either side of the mesh formed by the strings of a tennis racket, then move your hands. It will feel as though you have a cloth with a smooth pile like velvet between your hands. This is because your brain perceives a tactile perception that differs from what it should be.

You can experience such tactile illusions even without a tennis racket. Cut a round hole about 8 mm in diameter in a piece of card, place the index fingers of both hands against the hole, and move your fingers. It will feel as though you are touching a baby’s skin. Strangely, it may not feel as smooth if you use the thumb of one hand and the index finger of the other (Figure 1).

Figure 1. The baby skin tactile illusionThis experiential tool is a business-card-sized thick paper with a hole in it. The baby skin tactile illusion can easily be experienced at home if you cut out a hole 8 mm in diameter and touch it with both hands. At the presentation venue, he distributed cards printed on specially chosen paper, which already had the hole cut out.

Associate Professor Masashi Nakatani of Keio University’s Faculty of Environment and Information Studies explained this phenomenon at a presentation featuring reports on research funded by the Japan Science and Technology Agency’s (JST) Precursory Research for Embryonic Science and Technology (PRESTO) program.

“It’s possible that the brain processes somatic sensation in a different way when we touch an object with both hands from how it does when we touch that same object with only one hand,” he said.

He wants to investigate these differences to ascertain how haptic information is processed in the brain to produce sensation, and to link this study into research focused on perception mechanisms.

Such research presentations can, if creatively adapted, be delivered via online tools. However, Nakatani chose to rent an art gallery to present his results in the form of an exhibition.

“If it’s just about presenting the content of the research outcomes, one could do so online. However, online exhibitions are somewhat lacking compared with a physical experience of those who come in person to view the exhibition. I wanted to find a place where people could experience my research in real life during a lull in the COVID-19 outbreak,” he said.

“Physical experience” was a key phrase in his presentation. So why did Nakatani come to emphasize physical experiences to this extent after all his research into the sense of touch? One factor behind this was the mechanisms of touch and haptic impression.

After reviewing his activities so far, let us drill down to explore touch, haptic impression, and the experiences they create in greater depth.

Human fingers capable of distinguishing between vibration frequencies by touch

Bumpy, slick, slimy, smooth. One can gain a variety of haptic impressions by stroking an object with one’s fingertips. We can detect such information because we have four types of touch sensors —— called mechanoreceptors —— in our fingertips.

The four types of sensors each have a different role. The first, Merkel cell-neurite complex detects rough or bumpy surfaces. The second, sensory nerve endings in Meissner’s corpuscles, are sensitive to the faint slipping sensation created when tracing something with a finger.

The third, sensory nerve endings in Pacinian corpuscles, respond well to high vibration frequencies, resulting in sensations of smoothness and crinkliness. The fourth, sensory nerve endings in Ruffini endings, respond to the stretching of skin.

Touching an object with a finger and moving it causes faint vibrations. Like sounds, vibrations have frequencies. Just as the human ear can hear the difference between high-pitched and low-pitched sounds, the human finger can tell the vibrations of a rough surface from those of a smooth surface by touch.

There are mechanoreceptors sensitive to the various frequencies that occur between fingers and objects, enabling us to distinguish differences between bumpy and smooth surfaces. Not only these mechanoreceptors, but also the shape of our fingers help to detect these vibrations. The fingerprints amplify the vibrations that occur when we touch an object, enhancing perceptual sensitivity.

The sensation that results from touching something differs according to haptic explorations. The speed at which one’s hand moves and how hard one presses down on it also make a difference. The sense of touch uses such mechanisms to interpret the nature of an object’s surface.

In 2007, Nakatani and some fellow researchers launched an initiative called TECHTILE (TECHnology based tacTILE design) to explore the possibilities for expression based on tactile replication technology.

Since the Yamaguchi Center for Arts and Media (YCAM) subsequently became interested in the TECHTILE project, he undertook a two-year joint project with the center, resulting in the development of the TECHTILE toolkit (Figure 2A), which is an introductory kit for treating touch as expressive media.

Figure 2A. TECHTILE toolkitThe introductory kit for exploring the sense of touch, consisting of a microphone, amplifier box, and vibration module.

The TECHTILE toolkit is very simple. It consists solely of a voice microphone that converts air vibrations into electrical signals, an amplifier box that magnifies the electrical signals, and a vibration module that enables the electrical signals to be sensed as vibrations. Aside from the vibration module, which is the part held in the hand, the configuration is the same as an ordinary recording and playback device.

“It is known that the several genes are involved in ensuring that both hearing and touch functions normally,” Nakatani said.

Both hearing, which involves picking up vibrations in the air, and touch, which involves interpreting mechanical vibrations, are the same, in the sense that they both involve feeling vibrations that originate outside the body. Replicating the timing at which the vibrations are presented will enable touch stimuli to be sent electrically to another place.

He and his colleagues have presented examples of the TECHTILE toolkit’s use, employing it to transmit the vibrations of a marble rolling around inside a paper cup to another paper cup and amplify the fizziness of sparkling water poured into a cup to provide expressions that can actually be felt (Figure 2B).

Figure 2B. The marble experimentIt replicates tactile vibration in a different location by converting vibration information into electrical signals in an audio channel.

Haptic expression opens up possibilities for an even greater range of expression through the use of vision, hearing, and words.

“Touch can be interpreted in diverse ways. The information we pick up often has more than one meaning. This is why, when presented with information alongside other sensations, sense of touch can function as a sensory clue for understanding a physical experience,” he said.

In their book Shokuraku Nyumon [Introduction to Haptics —— Rediscovering the world through touch,] he and his colleagues state that the three elements of touch are the physical object, the body, and the mental imagery.

The material is the target being touched. The body refers to the part doing the touching and the way it touches the material. The mental imagery is a combination of the multimodal interactions from vision and hearing, the impression gained from words, and recall from previous experiences. In Japanese, onomatopoeia is often used to express mental imageries in words.

• *onomatopoeia : In Japanese, the general term for giongo, which are words expressing sounds and noises produced by objects and animals, such as “clunk” and “yap,” and gitaigo, which are words representing emotions and other states that do not make a noise, such as the words indicating excitement (“waku-waku”) and surprise (“bikkuri”). Japanese is said to be particularly rich in onomatopoeia.

Mental imageries differ even when the touch stimulus is the same

While we talk about research into touch as though it were something straightforward, the fact is that sensitivity to the same stimulus differs from one part of the body to another. And mental imageries include past memories, so there are substantial differences between individuals according to the recipient. With haptic impression —— that is to say, the impression of touch —— in particular, it is not possible to tell what is true.

In 2017, Nakatani launched a research lab at Keio University’s Shonan Fujisawa Campus. He chose physical interactions between parents and children as one of the research topics.

His recent presentation focused on parent-child interaction at different parts of the body. Although the act of touching is the same, the impression received differs according to whether it is the child or the parent who initiates the touch and which part of the body is touched. Sensory stimulus entering the brain via touch is interpreted after being blended with vision, hearing, and memories of past experiences to form a cohesive imagery.

“When we see a scene where parents and children physically interact with each other, we can’t tell from the outside the level of affection they share. If the purpose is just to provide personal care, physical interaction is not a sign of affection.” Nakatani said.

The impressions received in the mind from receiving exactly the same touch stimulus could differ. Research will be needed in the future to ascertain how much time modern caregivers have for touching their children in an affectionate way.

The catalyst for his decision to focus on the psychological effects of haptic impression was a student’s research into adding tactile feedback to text chat apps such as LINE by means of vibration. Having the device vibrate whenever a message was sent or received caused the user’s attention to focus on it every time.

“I realized there might be differences between tactile vibration for conveying special information and haptic information that’s a familiar part of daily life,” Nakatani said.

The act of touching has the potential to involve danger, as it brings the body into contact with objects. There is no problem if it is a safe environment, but when we are in the wilderness, we could sustain unexpected injuries if we touched everything we saw.

In early childhood, children check haptic feedback by touching the things around them and putting those things in their mouths. After such exploratory behavior mediated by touch, children become able to infer the heaviness or hardness or roughness of an object by sight or hearing, without touching it.

This is the reason why Nakatani began to focus on research into mental imageries resulting from physical touch. Touch is only a part of haptic impression.

The number of sensors in our body declines with age, affecting our sense of touch. However, we do not lose physical sensation itself as we age. In Japanese society, people are reserved about social interaction via handshakes and hugs. However, they are used a lot in some countries. The kind of physical contact seen in Western countries is a form of nonverbal communication for conveying emotion. Physical contact between parents and children is a means of conveying affection.

Four keywords were used during the presentation: guiding, moving, feeling, expressing. In undertaking the study, the team has relied on the assistance of people with infants aged between 8 and 12 months, who gain information via touch.

Using a 3D printer, the team makes objects with elements chosen from among commercially available toys to guide the infants. Using machine vision, they quantify the behavior demonstrated by the infants in their movements. This study helps in understanding the exploratory behavior demonstrated by infants through vision and touch.

Using the tactile illusions mentioned at the beginning, the researchers also study how the brain integrates information to feel smooth tactile perceptions like baby skin. In addition, they visualize and express the workings of a parent’s mind during parent-child interaction at different parts of the body (Figure 3).

Figure 3. Visualizing the reasons for touching/being touched evoked by touch on different parts of the bodyEven when the stimulus is the same, the sensation received differs according to who is doing the touching and who is being touched.
© Mao Midorikawa and Masashi Nakatani, SFC TOUCH LAB, Keio University 2019 | Faculty of Environment and Information Studies, Keio University

Set out in this way, one can see that haptic impression in childcare situations has an effect not only on the child, but also on the parent.

Nakatani, who has conducted research into the sense of touch in the context of childcare for the last three and a half years, spoke of the importance of a physical experience for haptic impression.

“What I’ve realized from considering research into children from the perspective of touch is the importance of increasing the number of a physical experience gained during childhood. A physical experience are vital to recalling past experience,” he said.

Linking a physical experience into the next experience is important

We gain haptic impression by superimposing the specific stimulus before our eyes with the mental imageries evoked by past experiences. Moreover, this does not end with a single physical experience; linking one physical experience into the next experience is important.

“Through a physical experience, it becomes easier for people to understand the principles behind phenomena, or they might develop an interest in similar things. I believe that it is easier to understand such repeated a physical experience through touch than via any other sense. That’s because repeatedly experiencing something makes it easier to spot differences for oneself,” he said.

Touch is used most often during childhood, but does not outwardly play an active role thereafter. However, this enigmatic sense unobtrusively sustains our lives and provides us with reassurance.

The sensations obtained when we touch something do not only have psychological effects such as these. They can give people a feeling of reality that help them to accept reality.

Immediately after the outbreak of the COVID-19 pandemic, there were news reports that bereaved families were unable to touch the bodies of their deceased relatives. It might be difficult to clearly come to feel that a family member has died if one cannot touch them, even if one understands this fact intellectually. I asked Nakatani for his views on how to solve such emotional dilemmas.

“I believe that the psychological processes brought about by touch play a not inconsiderable role in accepting someone’s death. Japanese funeral rites would seem to symbolize the need to touch something physical when remembering the deceased: we place a picture of them in a frame to give them a physical form and pray to a tangible monument in the form of a memorial tablet bearing their posthumous Buddhist name,” he said.

Even if we cannot comprehend the situation immediately, we spend time in the same space and sometimes pick these reminders up to look at them. This helps to create a physical experience. This process is very similar to that by which children learn from a physical experience during play. The act of touching has two faces: the direct physical stimulus gained from touch and the formation of mental imageries from haptic impression.

To understand the sense of touch and haptic impressions, it is necessary to look at not only physical touch, but also its effects on the mind.