Research ArticleMEDICAL ROBOTS

Intelligent magnetic manipulation for gastrointestinal ultrasound

See allHide authors and affiliations

Science Robotics  19 Jun 2019:
Vol. 4, Issue 31, eaav7725
DOI: 10.1126/scirobotics.aav7725


Diagnostic endoscopy in the gastrointestinal tract has remained largely unchanged for decades and is limited to the visualization of the tissue surface, the collection of biopsy samples for diagnoses, and minor interventions such as clipping or tissue removal. In this work, we present the autonomous servoing of a magnetic capsule robot for in situ, subsurface diagnostics of microanatomy. We investigated and showed the feasibility of closed-loop magnetic control using digitized microultrasound (μUS) feedback; this is crucial for obtaining robust imaging in an unknown and unconstrained environment. We demonstrated the functionality of an autonomous servoing algorithm that uses μUS feedback, both on benchtop trials and in vivo in a porcine model. We have validated this magnetic μUS servoing in instances of autonomous linear probe motion and were able to locate markers in an agar phantom with 1.0 ± 0.9 mm position accuracy using a fusion of robot localization and μUS image information. This work demonstrates the feasibility of closed-loop robotic μUS imaging in the bowel without the need for either a rigid physical link between the transducer and extracorporeal tools or complex manual manipulation.


Diseases of the gastrointestinal (GI) tract are numerous and severe, accounting for about 8 million deaths annually worldwide (1). They include cancers—such as colorectal, gastric, and esophageal—and inflammatory bowel disease (IBD), e.g., ulcerative colitis and Crohn’s disease (1, 2). High-grade bowel cancers are associated with high mortality (3), and chronic IBD conditions are associated with poor quality of life (4) and increased risk of further complications that include bowel cancer (5). Diagnosis and management of GI diseases are typically performed using optical flexible endoscopy (FE). This is the current gold standard despite numerous drawbacks and very few design improvements since its introduction (6). FE relies on a gastroenterologist, or operator, guiding the semirigid insertion tube of the endoscope through the mouth or anus while steering only its tip. This distal end contains a camera for inspecting the mucosa (surface anatomy) for areas of interest and an instrument channel for performing interventions (7). Advancing (pushing, pulling, and twisting) the endoscope by manipulating the proximal end results in tissue stretching that causes patient discomfort and, in extreme cases, can perforate the bowel (8). A further limitation of FE is that inspection of the small bowel is challenging and is not routinely conducted; it can be costly, long in duration, difficult to perform, and painful for the patient (9). This has motivated the development and use of capsule endoscopy (CE)—a disruptive technology in which a patient swallows a small diagnostic capsule (10). CE has the advantages of ease of administration, minimal discomfort, and the ability to transit the entire GI tract; however, capsules are limited by passive locomotion (via peristalsis) and cannot carry out important interventions (11, 12).

GI disease diagnosis under current practice can be onerous and time-consuming. Biopsies are invasive (introducing the risk of adverse events), incur additional costs such as increased workload (1316), and cannot be performed using CE. Many mucosal defects are readily identifiable, whereas others can be more subtle and obscure in appearance. Therefore, guidelines recommend multiple biopsies of tissue with abnormal appearance and, in some cases, normal appearance; the route taken is dependent on the disease under investigation (14). After collection, the tissue specimens are studied in a pathology laboratory, and a patient may be requested to return for a follow-up FE, adding further cost and prolonging disease treatment. In situ histology that is both noninvasive and facilitates rapid diagnosis would therefore be extremely beneficial (17).

Several imaging modalities have been investigated to enhance in situ capabilities of diagnostic devices that may expedite, or eliminate, postprocedural pathology. Examples include narrow-band imaging (18), autofluorescence imaging (19), Raman spectroscopy (20), x-ray imaging (21), computed tomography (CT) (22), positron emission tomography (PET) (23), magnetic resonance imaging (MRI), optical coherence tomography (24), and endoscopic ultrasound (EUS) (25). Of these modalities, EUS is safe and low cost and has the ability to generate higher-resolution cross-sectional images than CT, PET, or MRI; furthermore, it has already been demonstrated to provide accurate staging of GI malignancies, including rectal cancer (26, 27). Recent advancements in EUS technology have enabled the study of thin (<3 mm) bowel layers by using high-frequency (>20 MHz), and therefore high-resolution, ultrasound (US)—called microultrasound (μUS) (2729). This ability to accurately measure the dimensions of subsurface tissue anatomy is known to be beneficial, because an increase in tissue thickness can be indicative of inflammation and disease (27). Only recently has μUS technology been integrated into ingestible capsules (28, 30) that have been shown to generate US images that correlate closely with GI tissue histology (31), which offers the potential to perform EUS, and specifically μUS, in all regions of the GI tract.

Although these capsules have acquired μUS images in the small bowel, they are limited in their capabilities because they are passive and so subject to the environment for both acoustic coupling, via tissue conforming to the capsule, and locomotion, via peristalsis. This uncontrolled approach makes operator-led targeted diagnosis and intervention impossible. It also fundamentally limits US image acquisition in other areas of the GI tract, such as the stomach and colon, where the cavity is considerably larger than the capsule. In these situations, the capsule is less constrained by the environment, and acoustic coupling with the tissue cannot be guaranteed. The problem is compounded by the presence of air, which is an effective barrier to acoustic coupling. A robotic system that enables robust, targeted imaging in all locations in the GI tract by actively engaging with the environment could solve this. Furthermore, knowledge of the pose of the transducers could allow the sizes of lesions and their relative location in the GI tract to be identified. If implemented clinically, such a system offering targeted in situ diagnostics could reduce procedure time, improve pathology turnaround time, and enhance image-guided diagnostic yield, while also reducing physical and cognitive burden on the user.

The majority of previous work on robotic US can be grouped into three categories: (C1) robot-assisted probe manipulation, where robot motion is open loop with respect to US imaging (3236) and robotics is used to aid in teleoperation; (C2) visual servoing using US image feedback for the steering of a tool, where US is used as a method of tracking (visualizing) a robotic tool and aiding its navigation toward a target (3744); and (C3) visual servoing using US image feedback for the steering of the US transducer, where the US field of view can be actively, and even autonomously, adjusted (4549). The latter two categories involve the processing of US images and a robotic control system that operates in closed loop with respect to the US information. To the best of our knowledge, all previous work in this field has concerned the actuation of extracorporeal robots or robots that have been partially inserted into a human via rigid links, e.g., transrectal US robots (50). These robotic probes must maintain contact with the tissue to facilitate the transfer of energy in the form of US (51), are limited to a working range that is proximal to the anus, and thus are not feasible for navigating a fully mobile in vivo μUS transducer.

In this work, we investigated the feasibility of magnetically actuating a μUS probe in the form of a tethered robotic capsule endoscope (RCE). Magnetic actuation allows for the application of force and torque on the intracorporeal device and does not require a rigid-link connection between an extracorporeal actuation unit (i.e., permanent magnet or electromagnet) and an intracorporeal robot. Magnetic manipulation and μUS complement each other, with magnetic attraction inherently providing intimate contact with the tissue, which is crucial for reliable acoustic coupling. The μUS transducer signals can also be used in the robotic control loop to provide real-time feedback on the interaction between the RCE and its environment, supplementing the existing feedback modalities—vision, pose, and force—available for intelligent magnetic manipulation (5254). In this approach, we continuously acquire one-dimensional (1D) signals from a μUS transducer in the RCE, process these signals, and use the feedback in a magnetic actuation control loop. We have developed an autonomous control routine that facilitates the searching of a transducer configuration (i.e., RCE pose) that results in μUS image acquisition, without the user needing to learn to manually navigate the robot.

Our work contributes to the aforementioned categories C2 and C3 because we used US feedback to inform robot motion and used robot motion to reposition the μUS transducer, i.e., the RCE. This application combined US-based servoing of an in vivo device with embedded transducers, μUS-based servoing using 1D US signals, and the use of intelligent magnetic manipulation (i.e., probes actuated with a nonrigid link) to facilitate probe access to remote regions of the GI tract. This technology advances medical robotics, EUS, and GI endoscopy, with the following key benefits: (i) the addition of a level of environmental perception that can facilitate autonomous control (55), improving overall performance and usability; (ii) the ability to implement precise RCE, and thus μUS transducer, motions with the use of magnetic manipulation; (iii) a method of analyzing μUS image quality during operation and adjusting robotic control accordingly to ensure effective μUS image acquisition; and (iv) the potential to use this technology in the entire GI tract, given minor adjustments, or removal, of the tether.

The RCE system

To show feasibility of the proposed approach, we leveraged our previous work on intelligent magnetic actuation for painless colonoscopy (53) to develop a previously unexplored RCE and autonomous μUS robot control strategy. Our system (Fig. 1) consists of an extracorporeal permanent magnet (EPM) and an intracorporeal permanent magnet (IPM), the latter being embedded in the RCE. The RCE has magnetic field sensing capability that facilitates real-time localization of the device, the accuracy of which is about 2 mm and 3° (53). Throughout this manuscript, we assume this low-pass filtered localization feedback to be a ground truth. We use pose feedback for closed-loop magnetic control, where positions, orientations, and velocities of the RCE can be commanded. We note that the pose repeatability of the serial arm, and thus the EPM, is ±0.15 mm (LBR Med 14 R820, KUKA). This control resolution is thus the minimum bound on the control repeatability of the RCE. Environmental factors and field modeling have further effects on RCE repeatability. The forces and torques applied on the RCE are estimated using the point-dipole magnetic model. Using an on-board transducer and an oscilloscope, our system receives signal feedback in the form of waveforms. The logarithmic representation of these waveforms results in an A-scan. The waveforms are processed to develop an echo signal rating, or ESR. The ESR is an amplitude-based metric that determines the maximum magnitude of the received acoustic signal. It is computed by applying digital filters to a raw voltage output in digital form from μUS transducers. A high ESR for an acoustic waveform implies that high-amplitude echoes are present in the signal, and thus acoustic coupling and adequate alignment with the environment exist. The diagnostic task of determining the content of echoes is left to the operator. Our system can capture diagnostic images while the RCE moves and can use information from the A-scans to influence magnetic control. We use this feedback to autonomously capture μUS signals in an uncertain and unconstrained environment, where acoustic coupling cannot easily be confirmed. The acquisition of sequential A-scans facilitates the generation of B-scans, i.e., 2D US images with a logarithmic representation of echo amplitude. A graphical representation of our RCE moving through the colon lumen while building a B-scan is shown in Fig. 1A, and an overview of our system is shown in Fig. 1B. A video describing the overall concept can be found in the Supplementary Materials (movie S1).

Fig. 1 System description.

(A) A conceptual image of the in vivo RCE showing real-time acquisition of localized μUS images where ultrasonic features consistent with the histological layers of the bowel wall have been annotated. The RCE contains (a) μUS transducers, (b) an LED light source, (c) an irrigation channel, (d) a camera, (e) an IPM with circuitry that facilitates real-time pose estimation of the device, and (f) a soft, flexible tether. (B) The benchtop setup showing key components, as well as a closeup view of the RCE prototype.


We conducted a series of benchtop experiments to investigate the feasibility of this concept. It included (i) RCE control development, where tests were performed in a silicone phantom to better understand how to manipulate the RCE to acquire μUS images and characterize the relationship between control parameters and image quality; (ii) servoing using US feedback, where the feasibility of closing the robotic control loop using US echoes was explored; (iii) benchtop system validation, where the ESR method was validated on an acoustically realistic agar phantom (the demonstration of the system’s ability to autonomously acquire spatially relevant US images was also demonstrated during these tests); and (iv) in vivo validation and demonstration of an RCE acquiring μUS waveforms in a living porcine model.


RCE control development

We first characterized the robotic system by studying the relationship between RCE control parameters—i.e., tilt, roll, and RCE-substrate magnetic contact force—and μUS images. We found that the successful acquisition of μUS signals of sufficient amplitude by our RCE is sensitive to four main criteria—all with respect to the substrate: (i) the presence of an US coupling medium, (ii) transducer-tissue contact force, (iii) transducer tilt, and (iv) transducer roll. The conventions used for the tilt and roll of the RCE can be seen in fig. S1A. We conducted benchtop experiments using a silicone phantom (fig. S1B) to characterize this sensitivity and validate the system’s ability to effectively manipulate the transducers (i.e., RCE).

We conducted 10 instances of each test, with an operator teleoperating the EPM (i.e., serial manipulator): increasing coupling force from 0.25 to 2.5 N (RCE tilt = 0°), adjusting RCE tilt from −10° to 10° (coupling force = 0.6 N), and adjusting RCE roll in a span of 50° (coupling force = 0.6 N). We found that a higher force can strengthen US signals; however, the dependence of signals on coupling force is not linear or repeatable (Fig. 2A). We found that strong acoustic coupling exists in a ±3° tilt range of the RCE. In roll characterization trials, we found that acoustic coupling was strong in a roll range of about ±10.5°. These results show that contact force facilitates acoustic coupling (tight contact), and tilt and roll determine the transducer alignment, which is necessary to ensure that reflected echoes are successfully received.

Fig. 2 B-scans generated during RCE benchtop trials.

(A) B-scan images with the corresponding tilt and tissue-coupling force, collected during a force sensitivity test. The black arrow identifies a wall echo from our phantom. (B) Autonomous echo detection “echo maintenance” routine with three disturbances. The wall echo from the phantom is indicated by black arrows. The change in the observed depth of the back wall is attributed to variability in the phantom thickness, RCE contact force, RCE tilt, and thickness of coupling medium.

Sample-filtered time series of waveforms used for characterizing the force, tilt, and roll ranges are shown in Fig. 2A and figs. S2 (A and B) and S3A. The signal strength of μUS waveforms is subject to considerable attenuation and scattering without a coupling medium (in our case, we used water and US gel). For this reason, the RCE was designed with an embedded water channel for on-demand, in situ irrigation. When considering just the substrate, acoustic coupling is likely to be better in an in vivo setting compared with our silicone phantom, owing to the continuous secretion of mucus (56). A minimum contact force should be maintained, but the force characterization results suggest that attempting to characterize a force–echo strength relationship is likely not practical owing to a lack of robust repeatability; i.e., a known force does not necessarily induce a standard echo quality. Tissue contact force should also be kept to a minimum to avoid mucosal damage. A previous study has shown that no damage was inflicted on porcine mucosa when irrigation pressures of up to 3 bar were applied (57). The maximum (worst-case) pressure expected from our RCE is 0.75 bar. This was calculated by assuming an RCE tilt angle of 45°, a tissue indentation of 1 mm, and a maximum contact force achievable by our current system of 2.5 N. During the tilt and roll characterization tests and all other experimental work, the contact force did not exceed 1.5 N.

Experiments were subsequently performed to determine whether giving calibration-based pose commands and autonomously approaching them are sufficient for robust GI μUS imaging. We used the characterization results to determine the parameters that should be controlled and appropriate ranges for them. Tests were also performed to assess whether μUS signals could be successfully acquired while the RCE moves along a linear trajectory; these were conducted using both teleoperation and autonomous navigation. Table 1 summarizes this work.

Table 1 Autonomous RCE motion (open loop in μUS).

A summary of the tests carried out to assess the RCE’s ability to acquire μUS signals while in motion. Two configurations (A and B) were chosen to show the efficacy and repeatability of autonomous positioning for μUS acquisition in a defined environment. All the repetitions were performed on a horizontally oriented silicone phantom (fig. S1B).

View this table:

Although the RCE was able to reach desired configurations and traverse paths, reliable imaging was not always achieved. This was attributed primarily to the lack of μUS feedback and the necessity for both acoustic coupling and transducer alignment.

Servoing using US feedback

The acquisition of μUS echoes is dependent on successful acoustic coupling and alignment of the transducer with a substrate. Simply manipulating the RCE to a configuration or through a trajectory is insufficient for reliable μUS imaging (Table 1). Therefore, the ESR was used to provide a quantitative real-time measurement of the acoustic coupling; without this metric, a control loop could not be closed with μUS.

Our approach focuses on the amplitude of the received signal and compares it with an empirically set threshold (ESRthresh) at and above which echoes can be clearly seen. Although an ESR below ESRthresh may show potentially useful, but less clear, echoes, the threshold serves as a measure of the signal strength that the system strives to attain. We developed an autonomous echo-finding algorithm that uses this ESR for attaining clear acoustic images. The rating method and algorithm were evaluated during benchtop experiments. The user and autonomous system were first tasked with using the ESR to manipulate the RCE, adjusting tilt and force, to acquire echoes—i.e., echo detection—before the system’s ability to autonomously react to external disturbances—i.e., echo maintenance—was evaluated. Table 2 summarizes these tests and results. As seen in Table 2, the results suggest feasibility of our ESR method and its effectiveness as feedback for robust diagnostic imaging.

Table 2 Echo detection (closed loop in μUS).

A summary of the tests carried out to assess the efficacy of using μUS feedback. During echo detection, two substrate orientations were chosen to show repeatability and environmental adaptation. During echo maintenance, the external disturbances were in the form of pulling the tether and manually disturbing the pose of the EPM; these were included to simulate the presence of uncertainties that may be encountered in clinical practice. All repetitions were performed on a silicone phantom (fig. S1B).

View this table:

Benchtop system validation

The previous experimental trials focused on the development of robotic control and the method of closing the loop with μUS feedback. Further tests were performed to assess the performance of the system on a more acoustically realistic phantom (fig. S1C): an agar-based material with embedded features. Additional information on this phantom can be found in the Supplementary Materials. These tests demonstrate the capability to find echoes autonomously using the ESR method and to combine RCE localization and manipulation to complete an autonomous linear trajectory while gathering images with clear features. The latter demonstrates the acquisition of spatially relevant US images—i.e., B-scans with a spatially accurate horizontal axis. The results from these tests are summarized in Table 3.

Table 3 Benchtop validation tests.

A summary of the tests carried out to validate the ESR control algorithm and demonstrate the capability of using the system to create spatially relevant US images—performed on an agar-based phantom. To better mimic the uncertainty in in vivo conditions, the phantom was tilted by about 2°, unknown to the system.

View this table:

The echo detection results provide further support for the ESR method used. The teleoperated results were weaker than those seen on the silicone phantom, with the operator repeatedly failing to achieve reliable acoustic coupling. This was attributed to (i) the increased challenge of manipulating the capsule on a flatter, lower-friction substrate, where the curvature of the environment does not aid alignment, and (ii) the time limit set on the experiments, showing the increased cognitive burden of teleoperating the RCE (this requires the simultaneous monitoring and control of capsule pose, contact force, and ESR output with sufficient reaction speed). The spatial scanning results (with an example B-scan shown in fig. S4) suggest that it is possible to autonomously acquire spatially relevant US images by combining locomotion with autonomous echo detection—a capability that is necessary for future diagnostic tasks.

In vivo validation

The robotic system was evaluated in an in vivo porcine model (Fig. 3A), with the primary objectives of (i) successfully merging μUS, robotic, and magnetic technologies in vivo; (ii) comparing teleoperated and autonomous probe manipulations in terms of μUS image acquisition performance; and (iii) assessing the feasibility of performing targeted μUS for GI screening in a living organism. The experiments, summarized in Table 4, included both teleoperated and autonomous echo detection experiments, where the goal was to manipulate the RCE to acquire distinct in vivo μUS images of the bowel wall.

Fig. 3 Experimental setup and results for in vivo evaluation of our RCE.

(A) The in vivo experimental setup in the endoscopy suite. (B) A B-scan acquired during an in vivo trial and the corresponding ESR with the set threshold (red line). Features, characteristic of bowel wall layers, are annotated with black arrows. A deeper tissue feature is shown with a white arrow.

Table 4 Concept validation tests (in vivo).

A summary of the tests carried out to validate the concept, performed in a living porcine model. The autonomous echo detection tests were monitored by the clinician for safety reasons, but no assistance was given to the system.

View this table:

The level of electrical interference between the subsystems was acceptable, and the RCE integrity was maintained throughout the tests. Tissue from the animals was inspected postmortem, and no visual signs of trauma were seen. These results show that teleoperation was inadequate in this context; it was associated with high cognitive burden because of the required controlling of multiple variables simultaneously and in a complex, dynamic environment. The poor teleoperated results (Table 4) also show the importance of proper transducer alignment, because inadequate alignment resulted in poor acoustic coupling and fewer received echoes, despite there being continuous magnetic attraction (RCE-tissue contact). During autonomous operation, the robot was able to find an echo and react to a change in the ESR to briefly maintain the configuration at which the echo was acquired (Fig. 3B). A video summarizing the in vivo work can be found in the Supplementary Materials (movie S3).


With this work, we set out to answer the following fundamental questions: Can magnetic actuation be used to facilitate targeted μUS imaging of the bowel wall? What level of robotic assistance is necessary for robust μUS imaging? In doing so, our main contributions are (i) the development and in vivo evaluation of an RCE capable of targeted μUS imaging of the GI tract, (ii) the implementation of real-time μUS echo processing to enhance the robot’s environmental awareness, and (iii) the development of an autonomous control strategy that uses both magnetic field and μUS feedback for more robust μUS image acquisition. We have demonstrated that closed-loop robotic control using intelligent magnetic manipulation and μUS feedback is both feasible and advantageous, and it enables our RCE to acquire submillimeter-scale, localized images of the bowel wall with minimal input from the clinician. The successful benchtop and in vivo validations of autonomous μUS acquisition is crucial to determining the preclinical feasibility of the concept. The methods we applied in this work rely on μUS waveform feedback owing to its high resolution; however, the methods are relevant for other frequencies of US feedback. The core concepts are the use of an acoustic signal rating and magnetic servoing to achieve a clear acoustic image.

Magnetic fields not only allow safe and efficient transmission of force and torque through living tissue without the need for a rigid connection with the intracorporeal probe but also inherently lend themselves to the continuous transducer-tissue coupling that is crucial for μUS imaging. With the integration of magnetic control, our system is capable of precise, repeatable, and autonomous positioning of an RCE to visualize regions of interest. Precise manipulation is necessary, given the minimum contact force threshold and the small range of tilt (±3°) and roll (±10.5°) required for successful μUS signal acquisition. We demonstrated successful manipulation in both benchtop and in vivo environments where the RCE contact force and orientation were adjusted to achieve clear μUS signals. This demonstration of robotic μUS imaging in the bowel is important to support the need for further investigation toward enabling early in situ diagnosis anywhere along the GI tract.

Our method of echo detection was highly effective at quantifying the presence and strength of μUS echoes, and robotic assistance [a degree of autonomy as defined in (55)] was crucial for the robust acquisition of μUS images. The RCE, in an unknown in vivo environment, used μUS feedback as a sensing modality to both manipulate the RCE and “feel” the surrounding tissue and to gain clinically relevant histological information from the tissue in the form of μUS images of the bowel wall layers. Without real-time feedback, the user would not have known when coupling was achieved; autonomous echo detection would also have been technically impossible to implement. A compelling benefit of using closed-loop control in this context is robustness and repeatability when imaging the bowel wall in an unstructured environment. Teleoperation of our RCE proved to be challenging because no intuitive mapping between magnetic field input and US image quality exists. This resulted in a major cognitive burden because the operator was able to observe the ESR while attempting to maintain a contact force threshold and a narrow tilt range. Advancements in US array technology may reduce this sensitivity; however, the inherent limitation of manual effort to control many variables will still exist.

We found that approaching a location (using the on-board camera, for example) and then activating the autonomous echo-finding routine using the ESR was a robust and intuitive imaging method. On benchtop, we demonstrated that the RCE is effective at traversing linear trajectories while maintaining acoustic coupling. This enabled us to develop spatially relevant B-scans of an agar phantom with lateral spatial accuracy of about 1 mm. Our results suggest that our proposed methods scan large areas and produce images that facilitate observation and measurement of features. Closed-loop control was found to be essential, particularly in vivo, because it greatly increased usability and improved overall performance, with the results showing an advantage compared with teleoperation. The robotic system could perform the complex task of simultaneous magnetic and US control more effectively by handling the numerous control variables quickly and accurately.

Although the ESR index-based control worked successfully, we believe that the efficacy of our methods can be improved with a higher rate of μUS data acquisition. Our system was designed for preclinical feasibility and captured A-scans at less than 3 Hz. This resulted in cases where the RCE could not react fast enough to sudden interactions with obstacles in the environment. The data acquisition rate could be increased with a high-performance analog-to-digital data acquisition system, similar to that already used in the US literature (58). With a higher acquisition rate, the gradient of the ESR could be monitored, and the system could detect the continuous change in signal strength and allow for appropriate motion compensation. This would also allow the system to acquire data for reconstructed 3D images of the bowel wall by continuously collecting and merging localized μUS data as the RCE moves.

The current method of magnetic actuation does not enable RCE roll control about its central axis. It has been shown that magnetic control can be extended to all axes when manipulating the distribution of dipole elements in an actuated device (59); however, this comes with an increase in system complexity and a potential cost of implementation. As a simple solution, we chose to offset the internal magnet from the geometric center of the RCE, thus introducing a corrective torque; we also flattened the top of the RCE to increase roll resistance. These features result in a unique minimum energy configuration of the RCE in which the transducers are in contact with tissue. The most viable long-term and complete solution to this is to replace the single-element μUS transducers with curved μUS arrays and completely avoid the need for precise roll control of the RCE body, an approach that has been previously explored and shown to be feasible (60).

With achievable design modifications, the RCE could be suitable for use in any region in the GI tract. Two key areas for modification are the tether and the overall size of the RCE. A tether was included to simplify the on-board electronics and provide a fail-safe method to recover the RCE during in vivo tests. If the tether is made substantially thinner and more flexible—or if it is removed completely and replaced with wireless technology—the RCE could extend its reach to the entire GI tract. Wireless technology has already been implemented in both μUS (28) and magnetic actuation (52), where human studies have shown that a wireless magnetic capsule for gastroscopy achieves similar procedural accuracy to standard gastroscopy (61). To access narrower lumens, such as those in the small bowel, the RCE must be miniaturized, with the two most noteworthy size reductions being made by replacing the camera with a smaller alternative [e.g., 160K complementary metal-oxide semiconductor (CMOS) camera, 1.6 mm in diameter, FUJIKURA] and by reducing the diameter of the IPM. Although a reduction in IPM volume would reduce the applied magnetic wrench, this can be compensated with an increase in the volume of the EPM.

Upon collection of more US data and classification by experts, additional information will be gained about the environment and the clinical efficacy of using μUS for in situ diagnosis. Using methods such as deep neural networks (62), real-time generated A- and B-scans may potentially be processed on pretrained networks to quickly determine whether the transducer is in contact with tissue or debris or whether an echo pattern corresponds to healthy or diseased tissue (63). Our algorithm picks out maxima of acoustic waveforms, but additional information that artificial intelligence (AI) can more readily process is also available. We hypothesize that the relationships between physical landmarks, i.e., the histological layers of the bowel wall, and echo signals could be learned by methods such as convolution. This could result in an algorithm that characterizes the physical (i.e., acoustic) properties of tissue, rather than the properties of the waveform as is done in the ESR. Such an AI-driven approach may lend itself to autonomous diagnoses in the future. Additional processing of the B-scan images could also include feature tracking. Combined with higher data acquisition rates, this may allow microscale motion encoding and precise RCE pose control, e.g., for image stabilization.


The aim of the experimental portion of this work was to evaluate the new RCE technology and determine the clinical feasibility of conducting robot-assisted μUS imaging of the GI tract wall. The objectives of this study included the evaluations of (i) the interplay between our US system and the magnetic actuation system, (ii) the ability to magnetically manipulate the RCE while acquiring μUS images, (iii) the feasibility of using real-time processing of μUS images in feedback control, (iv) the feasibility of using this feedback to find echoes autonomously, (v) the ability to acquire spatially relevant images via simultaneous locomotion and echo detection, and (vi) in vivo performance to show preclinical feasibility. We begin this section with an overview of our hardware systems and provide a description of the data flows. We then elaborate our experimental methods.

System overview

The system (Fig. 4) consists of a custom-built RCE with embedded μUS transducers, a serial manipulator with a permanent magnet at its end effector for applying fields and gradients to the RCE, custom circuitry for magnetic localization and force and torque estimation, a μUS data acquisition system, and a custom software system on a desktop computer. Movie S1 provides an overview of the RCE concept.

Fig. 4 System schematic.

A diagram that summarizes the major components in the system and the data flow between them.

The RCE system

The RCE was developed and fabricated for in vivo use. It comprises a 3D-printed shell (Standard Clear resin, Form 2 3D printer, Formlabs), a camera (CMOS, MO-T1003-65-N01, Misumi), a light-emitting diode (LED) light source, an irrigation channel, a permanent magnet [cylindrical, axially magnetized, 11.11 mm in diameter and in length, neodymium iron boron (NdFeB), N52 grade, D77-N52, K&J Magnetics, USA] encased with localization circuitry [custom flex circuit (53)], and two miniature μUS transducers. The RCE was assembled by hand, sealed with epoxy (Permabond 4UV80HV), and coated with Parylene C for biocompatibility and lubricity (64). The current prototype, measuring 21 mm in diameter and 39 mm in length, includes a tether (6 mm in diameter, PEBAX material) for power and data transmission and as a fail-safe method to retrieve the device. The RCE is manipulated via the magnetic wrench between the small IPM and the larger EPM (cylindrical, axially magnetized, 101.6 mm in diameter and in length, NdFeB, N52 grade, ND_N-10195, Magnetworld AG, Germany) positioned at the end effector of a medical-grade, 7-DoF serial robotic manipulator (14-kg payload, LBR Med, KUKA).

The force estimation strategy relies on the recovery of RCE pose information. This is done using a localization strategy that generates 6–DOF, low-pass filtered, pose information at 100 Hz with about 2 mm and 3° accuracy (53). Throughout this manuscript, we assume that our localization reading is the ground truth. This assumption is justified by the validation of the method described elsewhere (53). The magnetic control method used in this work relied on linearizing magnetic wrenches and using them to compute necessary joint values of a serial manipulator; it is based on previous control work (53, 65, 66). Autonomous control routines in this study used a nonholonomic velocity controller for translation, where velocity was commanded along a trajectory and motion orthogonal to the path was treated as a position error. Orientation control was position level; i.e., specific angle tilts and pans were commanded as opposed to commanding angular velocities.

The control system was implemented using Robotic Operating System (ROS) (67). An overview of magnetic actuation is included in the Supplementary Materials. Vectors are described using bold lettering, v, and unit vectors are described using v̂. All points and vectors are represented in the global frame. The dipole model is used for magnetic control, and the underlying equations for the magnetic field, force (fm), and torque (τm), are described in the Supplementary Materials. Magnetic actuation of devices is achieved by modifying the direction and the gradient of an external field; these modifications result in changes in applied torque and force, respectively. This applied torque and force, or wrench, results in motion of the actuated device. The governing control relation for robot actuation is shown in Eq. 1; this expression is similar to previous works in the field (53, 68). This relationship is a linearization of the nonlinear relationship between magnetic wrench and EPM twist. The position and heading of the EPM are indicated using pe and m̂e, respectively, where me is the magnetic moment of the magnet. The implementation of proportional-integral-derivative control is denoted by pid(). The heading of the RCE is denoted by m̂i, and the use of the subscript “d” indicates a desired value that is commanded by a higher-level control. The heading error eh is computed using m̂i×m̂id, and the error magnitude is the angle between these vectors. The magnetic actuation Jacobian is Je ∈ ℝ6×6. The pseudo-inverse of Je is given by Je+[δpeδm̂e]=[fmpefmm̂eτmpeτmm̂e]+[δfmdδτmd]=Je+[pid(epxy), pid(fz), pid(eh)]T(1)

The desired EPM motion is then converted to joint-level commands for the serial robot using the relationship between EPM motion and joint commands [Eq. 2 (68)], where S() indicates the skew-symmetric form of the cross-product operation, the purpose of which is to compensate for the symmetry of the EPM’s field about its magnetization axis. The identity matrix is represented as I3, and a matrix of zeroes is represented by O3. The geometric Jacobian Jr is that of the serial manipulator[δpeδm̂e]=[I3O3O3S(m̂e)T]Jr(q)δq=JA(q)δq(2)

The changes in joint values are computed via Eq. 3, where ξ is a small damping scalar, integrated, and sent to the serial manipulator’s internal controllerδq=JA(q)T(JA(q)JA(q)T+ξI3)1[δpeδm̂e](3)

The tilt (α) and roll (γ) of the RCE are visualized in fig. S1A. The tilt is computed from Eq. 4. The roll is defined as rotation of the RCE about its major axis from a nominal orientationα=cos1(m̂iT(I3ẑ ẑT)m̂i(I3ẑ ẑT)m̂i)(4)

US transducers and system

We used two 5-mm-diameter PVDF (polyvinylidene fluoride) μUS transducers with a center frequency of 30 MHz and a physical focus at 6 mm (f# = 1.2) mounted on the top surface of the RCE with the faces embedded about 1.5 mm into the RCE to bring the focal point closer to the luminal surface of the bowel wall. PVDF was used instead of the more common piezoceramic because of its flexibility and transducer manufacturability; a 30-MHz transducer required a conductive film of 9 μm in thickness, making it easy to achieve a spherical focus in a small package. Data were acquired from one transducer, and a second one was installed for redundancy. The transducers were connected to a custom printed circuit board (PCB) inside the RCE. This PCB was connected to external hardware via individually shielded micro-coaxial cables (42 AWG core, 9442 WH033, Alpha Wire, USA) that provided some electrical shielding from the other RCE circuitry. The US transducers were pulsed during transmission, and the received echoes were amplified by a commercial pulser-receiver (DPR300, JSR Ultrasonics, Imaginant Inc., USA) with 70 dB gain before digitization with a two-channel, 70-MHz oscilloscope (DSOX2002A, Keysight Technologies). After digitization, signals were envelope-detected using a Hilbert transform and logarithmically compressed.

Autonomous echo-finding algorithm

We implemented a waveform processing algorithm in Python to compute the ESR. This ROS node interfaced with an oscilloscope using the Python Universal Serial Bus Test, the Measurement Class library, and a Keysight Technologies application programming interface (API) to acquire waveforms from the oscilloscope at a maximum rate of 3 Hz. Waveform processing consisted of (i) trimming the raw waveform (an array of voltages) to a desired starting processing depth, (ii) applying a forward-backward linear filter using SciPy, (iii) flattening the signal using a low-pass filter to make relative echo magnitudes appropriate, and (iv) computing the maximum peak-to-peak waveform amplitude range in the waveform, which is the ESR. This ESR is then filtered over time. The ESR is an amplitude-based metric that informs an operator, or computing system, of the presence of acoustic information in a signal. We note that the clinical interpretation of this signal is left to the operator. Because the metric is amplitude-based, the ESR is not highly sensitive to filter parameters. To demonstrate what the μUS images that are used to compute the ESR look like, we refer the reader to figs. S3 and S5B, which are time-series images of the filtered waveforms that correspond to those shown in Figs. 2 and 3B. Here, we emphasize the relative clarity with which an echo can be identified (when comparing Fig. 2 with fig. S3). Because the μUS transducer is used as both transmitter and receiver in the system, residual energy from the transmit pulse is seen in the form of “ringdown.” We therefore implemented an additional ringdown-compensation filter by subtracting a moving average of the ringdown portion of the signal at each time step as followspBr=pBi=1npBn(5)

Here, pBm is a vector that represents the waveform values (voltages) of length m. The number of waveforms that are used in the filter is indicated by n. The length of this vector corresponds to the depth of an A-scan. The ringdown-filtered waveform segment is referred to by pBrm. This filter assists with visualizing areas of interest at smaller tissue depths, which may be otherwise obscured by the ringdown feature. It is demonstrated in fig. S5, where fig. S5A shows the B-scan without the ringdown filter, and fig. S5B shows a time series of filtered waveforms combined with ringdown compensation. The waveforms in fig. S5B are used by our real-time algorithm to compute the ESR.

The echo-finding algorithm was implemented using a low US acquisition rate and aimed to show the feasibility of autonomous echo acquisition and optimization. Our autonomous method, illustrated in Fig. 5, continuously increments the desired tilt of the RCE while maintaining a constant, predefined, tissue contact force. The algorithm is activated while the RCE is in an arbitrary pose—with or without transducer-tissue coupling. The algorithm then goes to and maintains a force while tilting the RCE between hard-coded tilt limits. The control process obtains ESR updates from a separate echo processing script and continuously checks whether the echo is above a desired threshold (ESRthresh). The ESRthresh is determined before algorithm usage via a calibration procedure where an operator teleoperates the RCE until acoustic coupling is obtained. The value of the ESR that results in an adequate acoustic signal is then used as ESRthresh. This approach was used because of the high variability between acoustic characteristics of different substrates. Once the control process observes an ESR value that is above ESRthresh, the controller switches states from “searching” to “maintaining RCE configuration.” Here, the EPM makes motions that maintain the RCE in the pose in which distinct echoes were observed. In the case that the control process observes a loss of echo, it switches back to the “search” state (tilting routine).

Fig. 5 Block diagram control schematic of our RCE system.

The US system components are shown in yellow, the magnetic control system is shown in blue, and the auxiliary inputs and outputs are shown in green.

Benchtop testing

Benchtop testing was performed with the primary goals of characterizing and refining the system in a known environment. For characterization trials, a phantom tissue substrate, 200 mm in length and ~7 mm in thickness, was made from cast silicone (Ecoflex 00-30 silicone and 10% Slacker by mass). The phantom was placed in a semicylindrical acrylic tube (77 mm internal diameter) and attached to an adjustable frame. The choice of silicone and its thickness were made to allow the RCE to deform into the substrate while also allowing repeatable/comparable measurements. For validation tests, an agar-based formulation was used to develop a phantom that was acoustically closer to tissue (6971). It included fabric mesh to simulate features of convoluted shape, microscale particles to provide attenuation and speckle, and several regularly sized and spaced features along its length. A full description of this phantom can be found in the Supplementary Materials. Acoustic coupling gel was used as both a coupling medium and a lubricant. The individual performing the experiments had experience with operating the robotic arm and the construction of the RCE. The benchtop platform and a summary of the testing can be seen in movie S2.

Characterizing the effect RCE motion and force had on echo quality was done using the semicylindrical silicone substrate placed in a horizontal configuration. Given a known substrate orientation, RCE pose feedback provided the relative tilt between the transducers and the phantom. To ensure that echoes are most effectively reflected to the transducer, the transducer’s focus should be as close to normal to the substrate. Knowledge of transducer-tissue contact force, transducer tilt, and transducer roll sensitivity provides perspective as to which parameters of the RCE are important in acquiring and improving localized μUS images. Given that the magnetization axis of the magnet in the RCE is along its body, the RCE’s roll cannot be controlled. However, we placed the magnet off the center body axis of the RCE to induce a restorative torque to keep the transducers biased toward a vertical orientation (72).

The effect that force (RCE-substrate contact force) had on the μUS signal was characterized by conducting 10 force-varying motions while data were acquired from a μUS transducer; the force was increased from 0.25 to 2.5 N under teleoperation while tilt was kept as horizontal (i.e., 0°) as possible. To characterize the effect that RCE tilt had on the μUS signal, we used a similar protocol, but this time, the operator adjusted the RCE tilt between ±10° via teleoperation, whereas force was kept constant at 0.6 N. The roll characterization varied in protocol in that the RCE was rolled by manually applying a torque to the device. The roll angle was varied in a 50° range while force was kept nearly constant at 0.6 N.

Appropriate control strategies and parameter (force and tilt) values for acquiring μUS signals during RCE motion were chosen based on the results from the characterization work. In the first set of experiments, the RCE autonomously approached two configurations of force and tilt. The two configurations were chosen to show control repeatability and were (i) –2° tilt and 1-N coupling force and (ii) 3° tilt and 0.9-N coupling force. The phantom substrate was oriented horizontally. Before each repetition, the user placed the RCE into an arbitrary decoupled pose and force. The test was then initiated, and the system autonomously approached the desired configuration while μUS signals were continuously acquired. This autonomous maneuver can be seen in movie S2. Tests were done to assess the feasibility of moving the RCE through a linear trajectory while continuously acquiring localized μUS signals. The RCE was placed into a desired configuration (0° to 1° tilt and 0.6-N force) before being moved along an 8-cm linear trajectory under both teleoperated and autonomous control regimes. Both the system and user were tasked with maintaining the desired configuration during locomotion in an attempt to maintain μUS coupling without ESR feedback. Localization data could be combined with continuously acquired μUS signals to generate spatially relevant μUS images, i.e., 2D B-scans of depth and width/distance. Although our recorded position is 3D, the scans are plotted in 2D with respect to the distance traveled by the RCE.

An objective assessment of ESR was not conducted because a universal standard indicator for US image quality does not exist. The ESR was first assessed subjectively by visually confirming that increases in ESR correspond to improvements in echo amplitude seen in processed B-scans. Two test protocols were then used to assess whether the ESR could be effectively used as a form of feedback in the robotic control loop (these preliminary tests were done using the silicone phantom). In the first protocol, we conducted echo detection trials in both a horizontal phantom and a tilted phantom to simulate the unknown environment configuration that would be present during use in vivo. Both teleoperation and autonomous control were used. The RCE was first placed in an arbitrary start pose (transducers decoupled from substrate) before the tilt was adjusted within hard limits until ESRthresh was exceeded (i.e., μUS coupling achieved). A trial was considered successful if multiple distinct μUS echoes were captured. During the autonomous tests, once the RCE found acoustic coupling, it used magnetic closed-loop control to maintain its configuration; simultaneously, the system continued to check for a loss of acoustic coupling. The second protocol was conducted to observe whether the system could autonomously react to external disturbances and attempt to maintain μUS coupling. From an arbitrary start pose, the RCE was tasked with autonomously detecting an echo and then maintaining that configuration in the presence of disturbances. Disturbances were induced either by teleoperating the EPM to override the autonomous routine and cause unwanted motion in the RCE or by pulling on the tether of the RCE to decouple it acoustically from the substrate. Movie S2 shows this use of the ESR and an example of the disturbances used.

The acoustically relevant agar phantom was used first to validate autonomous echo detection (i.e., the ESR methodology) and, second, for additional μUS servoing experiments that demonstrated combined echo detection and autonomous locomotion. In the former, we repeated the echo detection tests performed on the silicone phantom under both teleoperated and autonomous control regimes (n = 10). In the latter, we conducted 20 closed-loop teleoperation trials where autonomous servoing was used. In 10 trials, the RCE traversed an area where large acoustic features were present. In the other 10 trials, smaller acoustic features were present. The travel distance (trajectory length) was 15 cm in all trials. We developed and validated a linear traversal strategy for the RCE where the system entered a μUS servoing mode when acoustic coupling was lost. Once coupling was recovered, the RCE attempted to maintain the orientation in which the high-amplitude echo, i.e., high ESR, was achieved. This methodology proved to be robust, and the majority of markers were identified. We were able to measure widths of markers and spacing between them with about 1-mm accuracy.

In vivo testing

The intended outcome of these tests was to show the core functionality (i.e., acquiring μUS echoes using a magnetically actuated RCE) and was necessary to determine whether this technology should be explored further. Before the in vivo trial, the capsule was coated with a conformal film of Parylene C using a vacuum deposition tool (SCS PDS 2010, Specialty Coating Systems, IN, USA). The surface was primed with A174 silane adhesion promoter before deposition. Parylene C is a USP Class VI polymer that is commonly used for its biocompatibility, lubricity, and moisture barrier properties (64). This coating provided an additional layer of protection between the capsule and the surrounding tissue. Adhesion testing of Parylene C to the resin and leak testing of the capsule were also conducted before the in vivo trial, as detailed in the Supplementary Materials.

The trial was performed at a dedicated large animal research facility in the Roslin Institute, University of Edinburgh. The study was conducted under Home Office (UK) License (Procedure Project License: PF5151DAF) in accordance with the Animal (Scientific Procedures) Act 1986. The choice of a porcine as a model is based on GI anatomy comparable with human (73, 74). This includes bowel dimensions, particularly lumen caliber and bowel wall structure. The trial was conducted with the porcine (British Landrace, female, 37 kg) placed under general, terminal anesthesia. The system was used in an operating theater that had access to a mobile fluoroscopy machine (“C-arm”, Ziehm Vision FD). The C-arm was only used for visualizing the RCE in situ and was not necessary for its operation. Before trial initiation, the bowel was cleaned gently with water to remove any residual fecal material, and the ESRthresh was calibrated in situ. This was the main source of feedback during tests and provided a quantitative measure of transducer-tissue contact to inform RCE motion.

The RCE was inserted per rectum and was advanced about 20 cm into the colon by a combination of pushing the tether and magnetic actuation. Air and water were administered through the irrigation channel when deemed necessary either to distend the bowel to advance the RCE or for improving transducer coupling. During teleoperated echo detection experiments, the RCE was first placed in a decoupled configuration. The test was then started, and the user increased the contact force and adjusted tilt within a predefined range until multiple distinct echoes were found or until no echo was found after more than 1 min of tilting motions. The RCE was then decoupled before completing the next repetition. During autonomous echo detection tests, the user placed the RCE in the colon and decoupled the transducer in the same way as the teleoperated experiments; however, in this case, the autonomous routine was started by pressing a button. The system then performed a search routine in which the RCE was tilted within a 10° range (−15° to −25°), and the force was increased. When an echo was detected, the system autonomously attempted to maintain the current configuration (tilting back and forth to compensate for environmental disturbances). If lost again, the search routine resumed. Movie S3 summarizes the in vivo work. Once the experiments were completed, the animal was euthanized, and necropsy was performed. There was no evidence of colonic perforation or gross trauma.


A description of the experimental data submitted with the manuscript

Fig. S1. RCE roll and tilt conventions and the benchtop phantoms.

Fig. S2. Characterization of tilt and roll ranges using time series of filtered waveforms.

Fig. S3. Series of filtered waveforms used to compute ESR during benchtop trials.

Fig. S4. Spatially relevant B-scan generation.

Fig. S5. Filtering of an in vivo μUS image.

Fig. S6. Agar phantom manufacture.

A description of the agar phantom and manufacture protocol

Table S1. Agar phantom composition.

A description of magnetic actuation methods

Parylene C coating and RCE leak testing

Data S1 (Zipped folder). All experimental data referenced in the manuscript.

Movie S1 (.mp4 format). Concept overview.

Movie S2 (.mp4 format). Benchtop testing.

Movie S3 (.mp4 format). In vivo testing.


Acknowledgments: We thank R. Mosses for effective management of the Sonopill Program, as well as the wider Sonopill team for their support and feedback during project meetings. We also thank R. McPhillips for help performing some tests, C. Lemke for help on data management and processing, and, last, J. McLaughlan for the use of laboratory equipment and advice on the fabrication of the agar phantom. Funding: This research was supported by the Engineering and Physical Sciences Research Council (UK) under the Sonopill Program (grant number EP/K034537/2), by the Royal Society (UK) under grant number CH160052, and by the National Institute of Biomedical Imaging and Bioengineering, USA, of the National Institutes of Health (award number R01EB018992). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Engineering and Physical Sciences Research Council or the National Institutes of Health. P.R.S. was funded by the NSF Graduate Research Fellowship Program under grant number 1445197. Author contributions: J.C.N. and P.R.S. worked together throughout the project and co-authored the paper. In the following description of contribution, “*” indicates (co-)leadership on a task: J.C.N.*, P.R.S.*, S.C.*, P.V.*, and K.L.O. worked on project conceptualization; P.R.S.* and J.C.N. worked on data curation; P.R.S.* and J.C.N.* worked on formal analysis; S.C.*, P.V.*, and K.L.O. worked on funding acquisition; J.C.N.*, J.W.M., P.R.S., H.S.L., B.F.C., R.E.C., and G.C. worked on investigation; J.C.N.* and P.R.S.* worked on methodology; J.C.N.*, S.C., and P.V. worked on project administration; S.C.*, P.V.*, M.P.Y.D., and R.E.C. provided resources; P.R.S.*, J.W.M., and J.C.N. developed the software; P.V.*, J.C.N., S.C., K.L.O., and M.P.Y.D. provided supervision; P.R.S.*, J.C.N.*, P.V., S.C., H.S.L., G.C., and B.F.C. were involved in validation; J.C.N.*, P.R.S., and J.W.M. worked on visualization; J.C.N.* and P.R.S* worked on writing (original draft); and J.C.N.*, P.R.S.*, J.W.M., K.L.O., H.S.L., B.F.C., G.C., M.P.Y.D., R.E.C., S.C., and P.V. worked on writing (review and editing). Competing interests: The authors declare that they have no competing interests. Data and materials availability: All experimental data referenced in this work are available in the Supplementary Materials (data S1). This folder also contains a Python script to assist with postprocessing.

Stay Connected to Science Robotics

Navigate This Article