MIT's ultrasound wristband opens a new phase in controlling robots and virtual environments
The idea that a person can use their own hand movements to control a robotic hand almost instantly has long belonged to the realm of laboratory demonstrations and technological prototypes that required cameras, bulky equipment, or sensor gloves. A new development from the Massachusetts Institute of Technology shows that this transition from the laboratory to more practical use could be closer than was thought until recently. Researchers at MIT have developed an ultrasound wristband, a wearable device that tracks hand movements in real time by imaging the muscles, tendons, and ligaments in the wrist area, and then, with the help of artificial intelligence, translates that data into the positions of the fingers and palm. The result is a system with which the user can wirelessly control a robotic hand, but also move objects in a virtual environment with a level of precision that has so far not been easily available in such a compact format.
The importance of such a breakthrough is best understood when one takes into account how biomechanically complex the human hand is. In everyday actions, such as scrolling on a smartphone screen, dozens of muscles, joints, tendons, and ligaments are involved. That is precisely why robotics and virtual reality systems have spent years trying to find a reliable way to capture fine hand movements, not only coarse gestures but also the transitions between them. According to the research team's description, the new device does not try to guess the user's intention only through electrical signals from the muscles, but literally “watches” how the internal structures of the wrist change during movement. This gives it a significantly richer set of information about what the hand is doing at a given moment.
How the device works and why it is different from previous approaches
The central part of the system is an ultrasound module built into a band approximately the size of a smartwatch. It continuously produces images of the internal structures of the wrist while the user moves their fingers and hand. These black-and-white ultrasound images are not by themselves enough for the robot to know whether it should squeeze its fingers, point with the index finger, or release an object. That is why the team also developed an artificial intelligence model that learns to connect changes in the ultrasound images with specific hand positions.
The researchers state that the thumb and fingers are capable of 22 degrees of freedom, that is, a large number of different directions of bending, extending, and rotating. In the ultrasound images of the wrist, they found regions that can be linked to individual movements. Some changes correspond, for example, to extending the thumb, others to movements of the index finger, and together they create a map from which the algorithm can reconstruct the overall position of the hand. For such a model to be reliable, it was necessary to combine ultrasound images with external motion capture. Volunteers therefore performed different gestures while surrounded by cameras, so the system learned to match what was happening inside the wrist with what was visible from the outside.
Such an approach differs substantially from the three most common previous solutions. First, there are camera-based systems, which can be precise, but do not work well in every space, are sensitive to hand occlusion, and often require carefully arranged equipment. Second, there are gloves with sensors, which can record movement, but at the same time limit the user's natural feel and movement. Third, there are wristbands or electrodes that read the muscles' electrical signals. Such systems are making progress, but their problem is environmental noise and lower sensitivity to very subtle changes. The MIT team believes that ultrasound imaging of the wrist can offer a more precise and more continuous insight into the mechanics of movement, precisely because it does not register only the surface signal, but the morphology of tissue in real time.
From a laboratory demonstration to a robot that plays and shoots hoops
The demonstrations presented by MIT are intentionally simple, but very effective because they clearly show what happens when a digital system “translates” the human hand faithfully enough. In one case, a user wearing the wristband controls a commercial robotic hand. When the person moves their fingers as if playing a keyboard, the robot follows those movements almost simultaneously and performs a simple melody on the piano. In another demonstration, the same robotic hand imitates finger tapping in order to toss a small ball into a tabletop hoop. The point of these experiments is not the music or the game itself, but the fact that the movements were transmitted wirelessly, intuitively, and without a conventional handheld controller.
The application in a virtual environment is equally important. The team developed a computer program connected to the wristband, so the user can enlarge and shrink a virtual object by pinching fingers, grasping, or releasing, and smoothly move it across the screen. In an era in which the virtual and augmented reality industry is looking for more natural ways of control, that part of the story may be the most commercially interesting. Classic controllers offer reliability, but rarely provide the feeling that the object is being controlled with one's own hand. Cameras again suffer from the limitations of space, lighting, and occlusion. Wearable ultrasound here presents itself as a possible compromise between precision, portability, and natural feel.
Why data collection is almost as important as the device itself
Although the demonstrations attract the greatest attention, the researchers place special emphasis on another goal: building a large database of hand movements. The current phase of the work includes collecting data from a larger number of users with different hand sizes, different finger shapes, and different styles of performing gestures. This is crucial because no control system will be truly broadly applicable if it works well only on a small group of people with similar anatomy.
The just-described study involved eight volunteers with different hand and wrist sizes. They wore the device while performing various gestures and grips, including the signs for all 26 letters of American Sign Language. In addition, they held objects such as a tennis ball, a plastic bottle, scissors, and a pencil. In all of these cases, the system, according to the authors, accurately tracked the position of the hand. Such data have a dual value. On the one hand, they serve to improve the wristband itself. On the other hand, they can become material for training humanoid robots that require ever more sophisticated manual dexterity.
This is precisely where the broader significance of the work becomes apparent. If a robot is to perform tasks in which fine control matters, it is not enough to teach it only “grab” or “release.” It is necessary to know how strongly to squeeze, at what angle to place the thumb, how to coordinate multiple fingers, and how to adapt the grip to an object of a different shape. For that reason, the MIT team believes that ultrasound wristbands like this could serve as a tool for mass collection of data on human dexterity, including high-precision tasks. In statements cited by MIT, the possibility is specifically mentioned that such datasets could one day help train humanoid robots for delicate procedures, including certain surgical procedures. Such claims, of course, should be read as a development direction, not as something ready for clinical use, but they clearly show the project's ambition.
What this development means for virtual reality, robotics, and medical technology
Xuanhe Zhao, a professor of mechanical engineering at MIT and one of the lead authors of the paper, assesses that this approach could have an immediate effect in replacing existing hand-tracking techniques in virtual and augmented reality. That assessment is important because it comes at a time when wearable technologies and human-machine interfaces are developing rapidly, but the market has still not received a solution that is at the same time sufficiently precise, sufficiently light, and sufficiently practical for everyday use. In a broader sense, the ultrasound wristband fits into MIT's multi-year research direction of developing wearable ultrasound. The same laboratory previously introduced ultrasound “stickers” for continuous imaging of internal organs and monitoring changes in deep tissues, which shows that this technology is not an isolated prototype but part of a broader platform.
This is important also because the boundary between medical technology and control interfaces is becoming increasingly blurred. In medicine, ultrasound is valued because it is non-invasive and safe. In robotics and virtual reality, its advantage is that it can provide deep insight into tissue movement without the need for an invasive procedure or optical systems that require direct line of sight. It is precisely this combination that opens up space for various applications: from rehabilitation and assistive technology to video games, industrial control, and remote operation of robots in sensitive environments.
However, between a convincing demonstration and a product for wide use, there are still a number of obstacles. The hardware needs to be further reduced in size, the algorithms need to be trained on a significantly larger number of movements and users, and the system must demonstrate reliability outside strictly controlled conditions. The question of adaptation among users is also important: how much individual “training” the device will need, and how well it will work immediately after being put on the hand. This is precisely why it is interesting that other research is also appearing in the field of wearable ultrasound, seeking to develop so-called generic systems, that is, those that transfer more easily between different users. This shows that the entire field is entering a phase in which the question is no longer only whether the technology can function, but also whether it can become robust enough for the real world.
Who is behind the research and where the next phase of development leads
The paper was published in the journal
Nature Electronics, and in addition to Xuanhe Zhao and Gengxi Lu, the authors include researchers from MIT and collaborators from the University of Southern California. Among the MIT co-authors listed are Xiaoyu Chen, Shucong Li, Bolei Deng, SeongHyeon Kim, Dian Li, Shu Wang, Runze Li, and Anantha Chandrakasan, while the external collaborators include Yushun Zheng, Junhang Zhang, Baoqiang Liu, Chen Gong, and Professor Qifa Zhou. Financial support was provided by MIT, the U.S. National Institutes of Health, the National Science Foundation, the U.S. Department of Defense, and the Singapore National Research Foundation through the Singapore-MIT Alliance for Research and Technology program.
Such a list of institutions and funders shows that the work lies at the intersection of several fields: fundamental science of materials and ultrasound, machine learning, robotics, and human-machine interfaces. In that sense, the wristband is not just another gadget, but an example of how wearable systems are developing toward ever deeper “reading” of the human body. Instead of a computer or robot relying only on what they see from the outside, there are more and more technologies trying to register the biomechanics of movement from within, but without surgery. This is a conceptual shift that could strongly influence the design of future interfaces in the coming years.
If development continues in the expected direction, in the future a user could wear a discreet wristband and use it to control digital objects, work tools, or robotic manipulators in a way that is closer to natural movement than to classic “command input.” For the virtual reality industry, this means more realistic interaction. For robotics, it opens the possibility of finer remote control and better training of humanoid systems. For the field of assistive technology and rehabilitation, it could mean a more intuitive interface for people for whom standard controllers are not suitable. And for the science of wearable ultrasound, it represents another confirmation that a technology long associated with hospital devices and doctors' offices is gradually finding a place on the user's body, in everyday movement, and outside the classic medical environment.
Sources:- MIT News – official announcement on the development of ultrasound “stickers” and wearable ultrasound as a foundation for newer control systems and tissue monitoring (link)
- MIT News – official announcement on the progress of MIT's ultrasound stickers for monitoring changes in deep organs, with statements from the research team (link)
- Nature – commentary by Chonghe Wang and Xuanhe Zhao on the state of and obstacles to the development of wearable ultrasound (link)
- Nature Communications – paper on a portable ultrasound armband as a controller for virtual reality, useful for the broader technological context of the field (link)
- Zhao Lab, MIT – profile of researcher Gengxi Lu and description of his work on bioadhesive ultrasound devices (link)
- MIT Technology Licensing Office – profile of Xuanhe Zhao and description of the laboratory's research focus at the intersection of humans and machines (link)
Find accommodation nearby
Creation time: 3 hours ago