Plato once remarked, “science is nothing but perception.” If this is true, then how might science based on one type of sensory experience benefit from the addition of another?

Human perception is based on input from all five physical senses, and in the imaging world, vision is paramount. Yet other forms of sensory perception are demanding their due as technology expands to meet the growing needs of doctors and researchers.

Consider ultrasound machines, which provide expectant parents with both visual and auditory cues about the development of their babies. By adding another perceptual layer to the experience, OB/Gyns are also able to provide a better standard of care.

At the Centre for Image Analysis at Uppsala University in Sweden, Erik Vidholm hopes to add an entirely new layer of depth and texture to the world of imaging. With stereographic imaging software and haptic devices, Vidholm is working to introduce an additional experiential quotient to the scanning world – touch.

A Useful Fraud

“In the end, all these technologies are all an illusion,” says Bruce Schena, DEng, an engineering fellow with Intuitive Surgical of Sunnyvale, Calif. “It’s a computer-generated magic trick to fool humans into thinking we’re interacting with something physical that isn’t there. It’s like a giant sleight of hand.”

Schena specializes in robotics and haptic technology. Much of his work involves helping computer systems approximate the human sensation of touch. “Humans are remarkable in their ability to detect fraud,” he says, “although there’s this willing suspension of disbelief if you do it well.”

Even if virtual sensations cannot be made to feel “real” enough, Schena says, human operators can learn to use them well enough to trust their results. Their value is derived from helping to quantify physical experience in another form.

“In the haptics world, we actually talk about rendering much like the visual world talks about rendering,” says Schena. “This is a display technology not unlike graphics or audio – a way of getting information from the computer to the person. Haptics can display these physical cues of weight and force back to a human operator.”

Schena has applied his knowledge of haptics to the field of surgical simulation. Among other pursuits, he was a principal contributor to the development of Intuitive Surgical’s da Vinci® robotic surgical system. According to Schena, most medical haptics research is being done in surgical simulation, making imaging a potentially useful technology partner, especially for surgical planning.

Likewise, in his current position with RaySearch Laboratories of Stockholm, Sweden, Vidholm conducts radiotherapy planning. Although he currently does not use haptics in this venture, he writes in his PhD thesis – “Visualization and Haptics for Interactive Medical Image Analysis” – that the union of the two sciences is not too far off.

 “The development of computer hardware and software has given [us] invaluable tools for performing these [image segmentation] tasks, but it is still very hard to exclude the human operator from the decision-making [process],” Vidholm says.

“Therefore, interactive or semi-automatic methods for image analysis and visualization are needed where the user can explore the data efficiently and provide his or her expert knowledge as input to the methods.”

Understanding how the science of touch can help expand the field of vision in the imaging world requires a working knowledge of haptics as a computerized sensory experience.

What Is Haptics?

According to Schena, the two haptic systems used to approximate the human sense of touch are force feedback and tactile feedback technologies. Force feedback, or proprioception, may be understood as muscle memory. It’s a kinesthetic concept that relies on the basic workings of bodies in space.

“Force feedback means using your body as both sensor and actuator – applying a force and knowing how hard you’re pushing on something,” says Schena. “You could put anesthetic on your fingertips and still know you’re picking up a 10-pound weight.”

Tactile feedback describes what most people commonly understand as the sensation of touch. In the human body, a network of receptors in and underneath the skin connects with the human nervous system, sensing and broadcasting differences in the surfaces of different objects.

This textural-sensory system detects variations as it vibrates and stretches, allowing the brain to distinguish between different structures – for instance, silk from sandpaper.

“As you move your finger across a surface, the grooves on your finger slip, slide, and grip, and kind of shear the skin in a way that your nerves are able to detect,” says Schena. “Mechanoreceptors can detect the differences between different grades of sandpaper just by moving your finger across them. It’s a very sophisticated way of sensing.”

Phantom Force

As far as imaging is concerned, all this work with haptics has the potential to help resolve the problem of slice segmentation.

Segmentation involves integrating the multiple slices, or data segments, captured by MR and CT devices into a fully formed image. This process is error-prone and not easily automated. Organs vary in shape and size, and differentiating between subtleties in neighboring tissues is difficult.

Vidholm’s hybrid approach marries stereographics with haptic technology to create a physical feedback that helps speed up segmentation. The technique was tested by matching scans with actual medical data; of course, none of it is possible in realtime.

“I have taken a first step toward more intuitive interactive segmentation methods, but there is much work left to be done before we will see these methods being used in practice,” Vidholm says.

The force feedback device with which Vidholm’s software was developed is a desktop stylus called the Phantom, manufactured by SensAble Technologies of Woburn, Mass. Devices like the Phantom, which generates resistance at a single point located at the stylus tip, are chiefly used to detect elastic properties, or differences in hardness, says Schena.

Vidholm used the Phantom to facilitate the placement of “seed points” inside the organ being scanned. These seed points allow the user interacting with the device to define or determine the boundaries of the organ – for instance, where the liver ends and the kidneys begin.

Then, from these points, the remainder of the shape of the organ can be extracted from various “region-growing” mathematical procedures.

As the 3-D shape of the organ fills out, the haptic information transmitted to the Phantom pen attempts to provide a different sensory experience that matches the contours of the organ in question. These cues help form a more accurate 3-D representation of the target image.

“The segmentation problem is probably the greatest bottleneck in terms of using CT or MR data for [preoperative] planning,” says Schena. “It’s a tough, slow, rather painstaking process. All these tools [Vidholm’s] developed to speed that up are valuable.”

According to Schena, Vidholm’s approach – 3-D haptics in a 3-D rendering environment – is an excellent division of labor between the human and computing worlds.

“I think this is probably the best combination of using humans for what they’re good for – pattern matching, identifying structures and subtleties – and the computer for wading through volumes of data quickly,” Schena says.

Possible Pitfalls

However, the challenges are significant, Schena says, even if they are clearly defined. A doctor doing preoperative planning with a virtual database may be able to palpate a virtual liver and detect the location of a mass, for example.

The imaging data would help to provide a touch-sensory interaction that provides resistance through the haptic device. What’s more difficult to approximate, says Schena, is the tactility component, or the actual “feel” of that mass.
 
“There are all kinds of ways to render environments with different physical properties,” Schena says.

“You build these 3-D objects; and you can assign them a low spring constant (squishy), [or] a high spring content (stiff). We’re good at generating forces and making the user feel like they’re holding an object, but we’re not really good at displaying the subtleties,” he says. “Texture-rendering is sort of crude at this point.”

Another key hurdle that emerges in the segmentation process is the convergence of datasets gathered from various imaging modalities. When information from multiple sources is combined, the pictures of the bodies in question change.

As Schena points out, organs move dramatically when the patient is sitting versus standing. This means preoperative planning can change from day to day because “things might not be where they were in the scan,” he says.

Soft tissue targets overlap, and that makes them just as difficult for scanning technology to distinguish as it is for live surgeons.

 “The kinds of cues that are in our physical world aren’t there in segmentation,” says Schena.

“Once you segment the tissue structures and the anatomy in one of these datasets, what you don’t get is any sense of the physical properties of these structures. The computer doesn’t know whether it’s hard as a rock or soft as a marshmallow. If you just look at the segmentation, it’s going to model everything equally.”

Schena acknowledges that although some properties of a scanned object, such as stiffness, may be measured by radiodensity, it may not necessarily be a fair approximation of how the properties of the tissue map to its physical structure.

“Many things have similar densities and dramatically different physical properties, like mercury and lead,” he says. “I think that’s not going to be solved by more computing horsepower.”

Neither are other tactile traits – for instance, slipperiness, adhesiveness, smoothness, or proximity to other tissue – easily mapped by imaging technology. “Temperature is another one,” Schena says. “Diagnostics are done through temperature; a tumor that has a lot of vasculature might be warmer than one that doesn’t have a lot of tissue.”

Another issue yet to be addressed is the limited nature of the haptic interaction afforded by a single point-of-contact device like the Phantom pen. As compared with surgical techniques like bowel sectionals, in which the entire hand acts as an input device, the Phantom’s single point of contact is highly limited.

“It’s kind of an inefficient way to render large datasets,” says Schena. “More single points would be useful, and then in time, the ability to render more surface information.”

Experience Is Fluid

Even with the technological challenges yet to be addressed by haptics research – even when considering the simplicity of systems available now relative to human sensory experience – Schena says these devices can teach professionals much about human perception and learning capacity.

“One of the things that’s really interesting about the human perceptual system is that the smallest amount of information in a new channel means more than you might think. The whole is greater than the sum of its parts," Schena says.

"It’s like watching animation without the audio track, then adding the audio track, then a haptic layer, then a tactile layer, [and finally] a smell layer. The benefit is almost geometric. The human brain is really amazing about fusing these modes of information, even if they’re not completely accurate. Even if the pipe is very small, it’s a valuable new addition.”

Schena continues, “What’s amazing is how adaptable we are. The haptics world is very much a display technology that is a channel of communication to the user. Even if it’s not coming from the normal place, it’s a cue, and the brain is great at adapting that. It’s pretty cool that these technologies are only decades old and we’re getting so much out of them.”

— Matthew N. Skoufalos is a N.J.-based freelancer. Direct all questions and comments to editorial@rt-image.com.