AI that reads and decodes brain waves: Operating machinery by the power of thought alone!

Operating machinery by the power of thought alone —— what sounds like technology out of a science fiction novel is starting to become a real possibility. Brain —— machine interface (BMI) is a technology that aims to make up for deficiencies in human neural function through the direct exchange of signals between the brain and a machine. Regarded as being particularly effective in promoting social engagement by patients with spinal cord injuries or amyotrophic lateral sclerosis (ALS), BMI could, in the future, potentially be combined with artificial intelligence (AI) and engineering technology to enable people to operate electric wheelchairs, powered exoskeletons, smart devices, home appliances, and all kinds of other devices by the power of thought alone.

A technology called brain–machine interface (BMI) has become the focus of global attention in recent years. BMI is a technology that aims to compensate for defective neural function in humans through a direct communication pathway for exchanging signals between the brain and an external device. In patients such as those suffering from the neurological disorder amyotrophic lateral sclerosis (ALS) or from injuries to the cervical or spinal cord, the brain still functions, but its instructions are no longer passed on through the body, preventing the individual from moving as they wish. Accordingly, it is hoped that BMI will improve the daily lives of such patients and facilitate their social engagement by measuring the waves of electrical signals emitted by the brain’s neurons (brain waves) and inferring what they wish to do, thereby enabling them to operate external devices.

Most research uses brain waves in the motor cortex and somatosensory cortex

BMI research has been ongoing for more than 20 years, but with clinical trials having begun in the U.S. and other countries after companies got involved, seeing it as a new field of business, there is now a prospect of the technology’s commercialization. In particular, last year’s launch of a clinical trial by a startup owned by notable entrepreneur Elon Musk made the headlines.

In BMI research and development, the core elements are brain wave measurement technology, brain wave decoding technology, and instrument control technology, underpinned by a foundation of neuroscience research. Thanks to prior research, we have learned various things about the brain activity that appears in brain waves. As different regions of the brain handle different functions, the area where brain waves should be measured varies according to which function is being observed. For example, the visual cortex, which is responsible for interpreting what we see, is located in the occipital lobe. The motor cortex, which controls movement, is found in a part of the frontal lobe called the precentral gyrus, within which the respective functions of everything from each of our fingers to our toes are restricted to specific areas (Figure 1). Right now, most research in the field of BMI uses brain waves in the motor cortex and somatosensory cortex, as the locations of functions in these have been identified.

Figure 1. Localization of function in the brainThe cerebrum is divided into multiple regions, each of which is responsible for a different function. Most research into BMI today uses brain waves in the motor cortex and somatosensory cortex.

Let me explain in simple terms how brain waves are decoded. When a person is in a relaxed state with their eyes closed, the waves in their brain are large and slow (alpha waves), with a frequency (number of waves repeated per second) of around 10 Hz. When they open their eyes and look at something, the alpha waves weaken and small, fast waves with a frequency of around 100 Hz (high-frequency waves) appear in a specific area of the visual cortex. Similar changes occur in the motor cortex when a person moves. If we can ascertain the part in which those high-frequency waves are occurring and what function that part controls, we can decode what the intention is behind the action they are trying to take.

However, as those high-frequency waves are very weak, it is not easy to measure them accurately. There are two ways to measure brain waves: a noninvasive method, which involves placing electrodes on the scalp, and an invasive method, which requires electrodes to be placed inside the brain itself. Although the noninvasive method is frequently used under normal circumstances in medical examinations, such as the diagnosis of epilepsy, application in BMI has revealed that this method is low in accuracy and also includes a lot of noise, so its use in BMI research today is a lot less common. High-precision brain waves can be obtained by means of an implantable BMI, which is an invasive method that requires surgery.

Accurate decoding of brain wave data is becoming possible with AI

Medical devices for implantation in the body originally date back to the development in the late 1960s of pacemakers, which provide electrical impulses to the heart to treat arrhythmia. Over the last 50 years or so, the use of such devices has gradually expanded to a variety of other organs —— one example is deep brain stimulation, in which electrical impulses sent deep into the brain are used to treat Parkinson’s disease. In the field of neurosurgery, in which we work, electrodes placed inside the skull are used in the clinical treatment of epilepsy, in order to accurately identify the region of the brain that becomes hyperexcitable during an epileptic seizure, so that surgical excision can be carried out. As it is necessary to excise only the portion where the anomaly is occurring, the accurate measurement of brain wave data is essential. This intracranial electrode technology is being applied to the development of BMI devices as well. Implantable devices take a lot of time and effort to develop, but they are a sustainable technology that can be used for decades, as in the case of cardiac pacemakers.

Thanks to recent developments in AI, brain signal decoding technology has progressed at a startling pace, with constant advances being made in achieving greater precision in decoding by training AIs on a vast quantity of highly accurate brain wave data. Robots and other engineering technologies are also evolving. While robots cannot freely carry out such a diverse array of movements as people can, they can demonstrate superhuman abilities when it comes to certain specific functions, such as carrying loads too heavy for humans. BMI research is achieving advances by integrating the latest innovative technologies, like these.

Designed for patients with severe ALS, the implantable BMI that we are developing is a device consisting of a sheet of 64 electrodes arranged at roughly 3 mm intervals and an electroencephalograph measuring 3×4 cm. The larger the number of electrodes and the narrower the gaps between them, the more detailed the brain wave measurement that becomes possible; with this device, we can deduce the movements of the entire arm. In a neurosurgical procedure, part of the skull is removed and the device implanted in its place, with the electrodes occupying the space between the skull and the brain, so there is no damage to the brain itself. The fact that the implant does not damage the brain is the most important point. Moreover, the device cannot be seen by others, as it is covered by the scalp after implantation.

The electroencephalograph is equipped with wireless data transfer and power supply functions. Brain wave data is transmitted to a computer via the wireless data transfer device; the AI then infers what arm movement the user intends to make and controls the relevant instrument. This use of wireless technology is another key point, as having data transmission and power supply wires exposed from the head would not only be an inconvenience in daily life, but also pose an increased risk of infection. Accordingly, we see wireless technology as being an essential function in BMI.

However, special amplifiers and electronic circuits are required to fully implant the device in the body and make it wireless, plus the implant needs to be small and low in power consumption. Circuit size and power consumption increase in proportion to the number of electrodes. The biggest problem is the fact that the device itself heats up when power consumption is high. As a sustained increase of 2°C in the device’s temperature within the body will be enough to cause burns to the tissue, ensuring that the implant has low power consumption is of vital importance to ensure patient safety.

Characters can be entered simply by visualizing them

Our first prototype was much larger in size than our current device, and had high power consumption and a low transmission speed. Over a dozen or more years, we have made numerous improvements and, due also in part to advances in wireless communications technology, we now have a device of a size, power consumption, and transmission speed suitable for practical application.

The main form of implantable BMI used in clinical research overseas uses exposed wiring connectors on the head and needle electrodes inserted into the brain. We believe that installing the device in an environment that is as closed as possible and measuring brain waves without injuring the brain is a safer approach that minimizes both the risk of infection and invasiveness.

What we aim to achieve with this implantable BMI, first of all, is control of assistive communication devices (Figure 2). Assistive communication devices that allow the user to enter characters using a switch or by moving their eyes are already widely used as communication tools by people with ALS and other severe disabilities. However, patients with progressive diseases such as ALS become unable to operate these devices once they can no longer move even their eyeballs.

Figure 2. Implantable BMIThe implantable BMI under development is aimed at providing ALS patients with support in everyday life by enabling them to operate an assistive communication device or robot with their brain waves. Wireless capability is a crucial function to ensure safety, as it reduces the burden on the patient.

If this implantable BMI is connected to an assistive communication device, while scrolling the characters on the screen, the user can enter a character by visualizing the clenching of their hand, for instance, when the character they want to enter appears. However, as entering characters using this method takes time, we are now developing a system that allows the user to employ the same flick input method used to type Japanese characters on smartphones.

In addition to assistive communication devices, we are also conducting applied research focused on robots. With the consent of patients who had electrodes temporarily implanted in their skulls as part of the aforementioned epilepsy treatment, we used brain wave data relating to their arm movements to control a robot. As a result, we found that we could make a robot’s arm move in a manner somewhat similar to human movements.

Furthermore, we are participating in a project entitled “The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint,” which is being undertaken within the Japanese government’s Moonshot Research and Development Program for a better future for society. Along with other research institutions, we are working on research into the use of implantable BMI to control cybernetic avatars (CAs). ALS patients lose the ability to express their emotions in their facial expressions as they become unable to move their facial muscles.

For example, while they might write “That’s interesting,” they cannot communicate that to someone else by means of their facial expression. However, they would be able to convey their feelings to others if they had an avatar —— a character that serves as a representation of the individual —— that could express emotions as it speaks. In addition, if the user is talking with someone about a film, for example, rather than simply using the word “interesting,” the CA would be able to add words from the conversation so far and say something like “That movie’s really interesting!”, which would facilitate smoother communication. If we could make this kind of avatar a reality, it would also lead to social engagement in three-dimensional virtual spaces (the metaverse) built on the internet.

Essential to the control of these avatars will be generative AI and large language models, which are the very latest AI technologies. Unfortunately, it is not currently possible to read emotions from brain waves alone, but AI can extrapolate from and add to text in the manner described above. Similarly, when it comes to human movements, when we go to grasp something, for instance, we do not consciously think about moving our fingers in a particular way to pick the object up. It is difficult to infer from brain waves the fine movements that we usually make without thinking. However, if we applied a robot equipped with AI functions, it would, for example, become possible for a user to have a robot arm reliably and securely grasp a plastic drink bottle in front of them simply by thinking “I want to pick it up,” and having the AI deduce their intention from their brain waves, combined with information from a camera.

Using electrical impulses to move arms and legs as one chooses

We believe that, by integrating BMI, AI, and engineering technologies in this way, it will become possible for people to operate all kinds of equipment using the power of thought, including electric wheelchairs, beds that can adjust a person’s position, powered exoskeletons, smart devices, and household electrical appliances. If this can be achieved, it will expand the sphere of daily life for people with disabilities, in both the virtual world and the real world (Figure 3).

Figure 3. A BMI can be linked to various devices and forms of ITIntegrating BMI, AI, and engineering technologies will expand the sphere of daily life for people with disabilities, in both the virtual world and the real world. There is also a possibility that people will be able to use BMI to control nerve impulses, thereby enabling them to move their own bodies.

Currently, we are at the stage of checking its safety in animal research, with the aim of launching clinical tests to verify that implantable BMI can be used to operate assistive communication devices. We have also gone a step further, embarking on research and development focused on a BMI-controlled system that stimulates nerves. The intended users of this system are people who have disabilities that affect their arms and legs, such as patients with ALS, spinal cord injury, or paralysis following a stroke. In this system, electrodes that provide electrical impulses will be placed at the site of arm and leg nerve bundles; BMI will then detect signals from the brain and the electrical impulses will move the individual’s arms and legs as they wish. This is a completely new concept of BMI.

As a result of conducting research into the brain for so many years, I have been struck afresh by the feeling that humans have immeasurable capabilities. I would have to say that the day when BMI will be able to perform these innumerable functions is still a very long way off in the future. However, even if we can replicate even just one lost function with the aid of BMI, it will, I believe, be tremendously meaningful. To bring this goal to fruition, in 2020 we established a startup company with the aim of manufacturing and selling implantable BMI devices, but financial resources remain a major challenge. Unlike the U.S., where the scale of venture capital investment is reportedly around 10 times the level here, it is difficult to turn innovative medical devices into a business in Japan. However, I am proud of the fact that our technology is capable of delivering very safe BMI at a low cost. I want to provide the rest of the world with our innovative BMI from Japan.

(Figures courtesy of Masayuki Hirata)