Research ArticleSPACE ROBOTS

AEGIS autonomous targeting for ChemCam on Mars Science Laboratory: Deployment and results of initial science team use

See allHide authors and affiliations

Science Robotics  21 Jun 2017:
Vol. 2, Issue 7, eaan4582
DOI: 10.1126/scirobotics.aan4582

Abstract

Limitations on interplanetary communications create operations latencies and slow progress in planetary surface missions, with particular challenges to narrow–field-of-view science instruments requiring precise targeting. The AEGIS (Autonomous Exploration for Gathering Increased Science) autonomous targeting system has been in routine use on NASA’s Curiosity Mars rover since May 2016, selecting targets for the ChemCam remote geochemical spectrometer instrument. AEGIS operates in two modes; in autonomous target selection, it identifies geological targets in images from the rover’s navigation cameras, choosing for itself targets that match the parameters specified by mission scientists the most, and immediately measures them with ChemCam, without Earth in the loop. In autonomous pointing refinement, the system corrects small pointing errors on the order of a few milliradians in observations targeted by operators on Earth, allowing very small features to be observed reliably on the first attempt. AEGIS consistently recognizes and selects the geological materials requested of it, parsing and interpreting geological scenes in tens to hundreds of seconds with very limited computing resources. Performance in autonomously selecting the most desired target material over the last 2.5 kilometers of driving into previously unexplored terrain exceeds 93% (where ~24% is expected without intelligent targeting), and all observations resulted in a successful geochemical observation. The system has substantially reduced lost time on the mission and markedly increased the pace of data collection with ChemCam. AEGIS autonomy has rapidly been adopted as an exploration tool by the mission scientists and has influenced their strategy for exploring the rover’s environment.

INTRODUCTION

Planetary exploration

Robotic planetary exploration missions typically focus on reconnaissance of a planetary surface or environment and on answering specific scientific questions by gathering data with planned observations. Traditionally, both of these goals are achieved through a process of exploration whereby data are collected by the robotic spacecraft and sent to Earth, where experts interpret those data and use them to generate new strategies and select new observations to further their understanding of the site. These new observations are sent to the spacecraft as commands, and the exploration process iterates in this way.

Modern planetary missions have, in many cases, become data-limited in their planning—that is, they are capable of gathering more data than they can send to Earth. This is a consequence of the large interplanetary distances (with resulting low data transmission rates) and growing capabilities of instruments (which produce ever greater volumes of data). Data-limited mission planning can lead to substantial time costs, especially when one-way light times or planetary dynamics lead to long planning latencies.

These time costs can be reduced by challenging the assumption that interpretations and decision-making must take place on Earth. “Science autonomy” is the capability of a robotic system to choose for itself which scientific data to collect or, having collected it, which data to send to Earth. It is distinct from other forms of spacecraft autonomy—for example, in management of onboard power usage or in navigation. In this paper, we present a new operational capability in science autonomy for Mars surface exploration. The AEGIS (Autonomous Exploration for Gathering Increased Science) software suite now allows autonomous target selection and data acquisition for the ChemCam instrument aboard the Mars Science Laboratory (MSL) mission’s Curiosity rover.

Mars Science Laboratory

NASA’s flagship MSL mission (1) landed its Curiosity rover in Mars’ Gale Crater in August 2012. The mission’s primary goal is to search for and characterize a past habitable environment on Mars and, to that end, carries an extensive suite of scientific instruments to investigate and map the nature and stratigraphy of the crater’s stratified central mound, Aeolis Mons. Now in the extended mission phase of its surface exploration, the mission has identified habitable fluviolacustrine environments (2) and extensively characterized the sedimentary and igneous materials, soil, sand dunes, and atmosphere at its landing site [e.g., (317)]. These investigations are carried out using a cyclical mission operations process (18) in which images and other observations from the rover are acquired on Mars, the data are returned to Earth and assessed by the science and engineering teams, further measurements and activities are carried out at the rover’s current position using the images sent back to select targets, and then the rover is commanded to drive into a new area and send back images for the next iteration of the process.

ChemCam

ChemCam (short for “chemistry and camera”) is a mast-mounted, targetable, remote-sensing instrument aboard the Curiosity rover (19, 20). It includes a laser-induced breakdown spectrometer (LIBS) capable of providing geochemical spectra on targets at ranges of up to 7 m and a context camera, the Remote Micro-Imager (RMI), which provides images of the LIBS targets, or of other targets at an unlimited distance. ChemCam has acquired more than 440,000 LIBS spectra in more than 14,000 observations of more than 1500 targets on the Martian surface since the surface mission began in August 2012. The instrument has been used extensively throughout the surface mission (21), and its data have contributed to many of the mission’s scientific insights [e.g., (2233)]. A view of the instrument and examples of its use on Mars are shown in Fig. 1.

Fig. 1 ChemCam.

(A) The ChemCam aperture is the large, circular mirror on the Curiosity rover’s Remote Sensing Mast (image credit: NASA/JPL-Caltech/MSSS). (B) The rock at the “Windjana” drill site was measured by ChemCam LIBS, leaving visible marks both on the surface (upper right) and inside the 16-mm-diameter drill hole (center) (image credit: NASA/JPL-Caltech/MSSS). (C) Soil targets are also measured, as in the RMI view of “Sandy_Dam” (image credit: NASA/JPL-Caltech/CNES/CNRS/LANL/IRAP/IAS/LPGN). (D) The RMI’s infinite-distance focus capability allows detailed views of distant targets, such as this area in the Peace Vallis alluvial fan, ~25 km away (image credit: NASA/JPL-Caltech/CNES/CNRS/LANL/IRAP/IAS/LPGN).

ChemCam’s ability to quickly survey the geochemistry of an area through standoff spectroscopy has been proven to be useful both directly for science investigations and as a tool for planning (21). ChemCam spectra can be used to select whether and where to make use of other instruments requiring greater time and other resources, such as those mounted on the rover’s robotic arm. Given its narrow field of view, and the requirement that the laser be focused to effect a LIBS measurement, ChemCam activities are typically targeted, with operators on Earth selecting targets using previously returned imagery.

Motivation for autonomous targeting

Because mission planning cannot occur in real time (due to light-time latencies, line-of-sight accessibility to the rover from Earth, and the motions of the planets and of relay spacecraft), there are times when remote sensing is limited by a ground-in-the-loop constraint. For example, after an early-afternoon drive into a new area, hours of good lighting conditions and sufficient rover power resources are often available, but targeting images have not yet reached the team on Earth. If the rover is to drive again the next sol (a sol is a Martian solar day, with a length of 24.6 hours), there may be very limited opportunity to observe targets at this location before the drive. Autonomous targeting for remote-sensing instruments allows some of the objectives at this new location to be achieved immediately postdrive, freeing up time in the ground-targetable period before the next drive. This time is so valuable that the ChemCam team innovated a blind targeting technique early in the mission, firing the LIBS postdrive at whatever rock or soil fell into a standard field of view beside the rover (34). An intelligent targeting system, which could select targets of the type preferred by the science team, would greatly increase the use and science return from this postdrive period and other similar phases of the mission.

Motivation for autonomous pointing refinement

Despite the highly precise pointing ability realized by MSL on Mars, there are limits to pointing accuracy that affect the results from ChemCam. Motor backlash, thermal effects, rover settling or shifting, limits in targeting image resolution, and limits in stereo computation can all be partially mitigated but nonetheless contribute to uncertainty in targeting of about ±2 mrad (21). Because ChemCam LIBS rasters typically have a point spacing of 1 to 2 mrad and a span of 2 to 20 mrad, this targeting uncertainty is significant, and for small surface features (such as narrow veins in a rock outcrop), it can mean missing a feature altogether. Such a miss can mean losing the measurement altogether (if the rover is to drive away soon after) or losing a day of Mars surface time (if the drive is delayed to make a second measurement attempt). An intelligent targeting system capable of recognizing fine-scale features and correcting pointing errors on the order of a few milliradians could prevent these losses.

AEGIS software system

The AEGIS software system was originally developed for the Mars Exploration Rover (MER) mission and has been operational on the Opportunity rover since 2010 (35). The system allows the rover to autonomously select targets in images from its navigation cameras (NavCams) and acquire follow-up images of them with its narrow-field science camera, the panoramic camera (PanCam). This capability allows images to be obtained mid-drive, or postdrive, when the rover is in a new position and the details of its environment are not well known. As compared with waiting for ground-in-the-loop targeting, this can save one to several days of mission time.

AEGIS has been adapted for the MSL mission to allow autonomous target selection and data acquisition for ChemCam. The system is capable of finding targets in images from either MSL’s NavCams or the ChemCam RMI. In the NavCam case, the operation is analogous to the use on the MER mission, for finding targets in unknown terrain after a drive into a new locale.

In the RMI case, because the RMI itself is narrow-field (20 mrad diameter) and requires accurate range for focus, AEGIS is used to refine pointing to fine-scale features, such as narrow rock veins. This RMI pointing refinement capability can be run not only on RMI images targeted by operators on Earth but also on RMI images acquired by AEGIS-on-NavCam targeting. The pointing refinement capability is therefore also available on targets not yet seen by operators on Earth.

AEGIS rollout to MSL

AEGIS was uplinked and installed in Curiosity’s flight software in October 2015 and then tested and validated over several months. It was approved for initial science utilization in May 2016, allowing the mission team to use it for autonomous targeting and pointing refinement when desired; the first such use occurred on mission sol 1343. This paper describes that deployment and the results from the AEGIS operational use so far. An enhanced rollout, with additional flexibility in source image pointing, target type, and number and nature of follow-up observations, is in development. The uploaded AEGIS system represents an addition of just over 21,000 lines of code to the 3.8 million lines in the MSL flight software.

RESULTS

Summary of AEGIS use by the MSL Science Operations Team

From sol 1343 to sol 1663 (representing activities planned on Earth between 13 May 2016 and 7 April 2017), AEGIS has been used 54 times, selecting 59 targets and observing each with ChemCam. Of these, two instances were for pointing refinement by analyzing targets seen in a ground-targeted RMI image. The remaining 52 instances were postdrive autonomous target selection activities using NavCam source images. Of these, 47 included commands for ChemCam observations on the top-ranked target found by AEGIS; the other 5 commanded observations of the top two targets found.

In every AEGIS run, the system has analyzed the geological scene in the source image (RMI or NavCam), detected feasible targets, ranked them, and selected a specified number of targets (top 1 or top 2). It has then obtained a full ChemCam observation suite (30 LIBS laser pulses and corresponding spectra each on nine observation points, with pre- and post-LIBS RMI images). In every case, AEGIS has adhered to safety constraints, including the following:

(1) Filtering out targets not within the safe LIBS firing distance

(2) Avoiding pointing the within the risk zone around the Sun’s path through the sky (which can damage ChemCam)

(3) Avoiding targeting the rover itself (which can damage the rover)

No AEGIS run has resulted in a failure to find targets or in poor-quality ChemCam data.

Selected results from the 52 NavCam source AEGIS runs are shown in Fig. 2, and results of both RMI source runs are shown in Fig. 3.

Fig. 2 Examples of AEGIS autonomous target selection.

In each, the NavCam field of view available to AEGIS is shown. All targets found by AEGIS are outlined; blue targets were rejected in filtering, and red targets were retained. The top-ranked target is shaded green, and the second-ranked target (if measured) is shaded orange. The NavCam images have been contrast-balanced. Note that objects whose contours intersect the edge of the image are rejected as potential targets by AEGIS. Left to right, top to bottom: Sols 1400, 1417, 1449, 1481, 1516, 1605, 1636, and 1660.

Fig. 3 Examples of AEGIS autonomous pointing refinement.

In all cases, the RMI field of view has a diameter of 19 mrad. Note that the spacing of the 3 × 3 grid of LIBS points is 1 mrad, and the pointing accuracy of the rover’s mast is ±2 mrad. Mosaics include pre- and post-LIBS RMI images. (A) Sol 1424 source image, targeted by the team on Earth using NavCam stereo information. Initial pointing marked in red. (B) Sol 1424 pointing refinement result; LIBS spots marked in red on mosaic of RMI images. Three of the nine points hit the desired bright feature. (C) Sol 1463 source image. (D) Sol 1463 pointing refinement result. Credit for all images: NASA/JPL-Caltech/CNES/CNRS/LANL/IRAP/IAS/LPGN.

Run times on the MSL rover flight computer

AEGIS runs on the Curiosity rover’s main flight computer, using a RAD750 processor (36) running at 133 MHz; the system has access to 16 megabytes of random-access memory. Time is precious on Mars, so use of autonomous systems relies on them carrying out their tasks reasonably quickly, so as to not compete for time with other science observations.

The autonomous pointing refinement process took 94 and 96 s to run in the two instances; this includes RMI image acquisition, processing, and analysis to find and select geological features, and the duration is consistent with tests in flight before rollout to the science team. For the autonomous targeting, which includes NavCam stereo image acquisition, processing, analysis, as well as computation of stereo distance to each target and checking targets for sun and rover body exclusion zones, followed by selecting targets, the process takes 150 to 450 s, varying nearly linearly with the number of targets found.

Increase in data return from the ChemCam instrument

As a result of the regular use of AEGIS for autonomous targeting, the postdrive period on many sols has become available for ChemCam observations. This has resulted in a significant increase in the rate of ChemCam observations carried out on Mars. A graph of this, expressed in cumulative LIBS shots over mission time (in sols), is shown in Fig. 4.

Fig. 4 AEGIS increases the rate of ChemCam observations.

A notable increase in the pace of ChemCam measurements occurs after AEGIS deployment on sol 1343.

Value of intelligent targeting over blind targeting

Although blind targeting is valuable for gathering more data, the potential advantage of intelligent targeting is the ability to obtain more data on preferred target types. AEGIS, in the initial rollout period since sol 1343, was enabled with one “target profile” per camera. For NavCam, AEGIS was set to prefer light-toned bedrock corresponding to common materials seen over the recent portion of the rover’s traverse.

AEGIS target profiles are realized by combining a variety of computer vision parameters for finding targets in an image, filtering, and ranking those targets. An initial profile, developed on the basis of terrain seen over the several kilometers of driving before rollout, was uploaded to MSL ahead of the first use by the science team. Results from field use on Mars allowed this initial profile to be refined and adapted; version 2 was first used on sol 1400 and remains in use as of sol 1662.

In both versions of the NavCam target profile, the intent was to find patches of bedrock (as opposed to loose float rocks, pebbles, and soil). Figure 5 shows AEGIS’ performance at choosing this preferred target type from sol 1343 to sol 1662. Also included are the results from modeling blind targeting for the same scenes, as a baseline for absence of intelligent targeting capability. The ChemCam blind targeting activity was aimed at a specific rover-relative angle, which allows reconstruction of what would have been seen by ChemCam had blind targeting been used instead of AEGIS intelligent targeting. For this, we have assumed a 10-point linear LIBS raster with 2-mrad spacing, typical of blind target observations.

Fig. 5 AEGIS intelligent targeting compared with blind targeting.

AEGIS target selection results. The target profile was set to prefer light-toned outcrop targets and was refined on sol 1400 based on the early in-service experience. “Rock” means float rock. The material listed here is what was measured by ChemCam LIBS; “Outcrop and Sand” implies that some LIBS shots fell on outcrop and some fell on sand.

AEGIS’ intelligent targeting capability reliably obtains measurements of the preferred outcrop material, far more often than the (essentially random) blind targeting. AEGIS placed all the LIBS points on outcrop for 44 of the 47 targets measured since the parameters were updated; blind targeting would have done so only 10 times and could only have measured 42 targets (the blind target pointing is fixed, but AEGIS can select multiple targets if resources allow).

Although soil measurements are also valuable, outcrop targets are generally prioritized for their value to understanding the geologic history of the environment. Blind targeting can still have uses in rover science operations (37), but it is no longer available for ChemCam, owing to a failure of the instrument’s autofocus capability (which has been recovered, but in a form that does not permit blind targeting). AEGIS is, therefore, now the sole means for making LIBS measurements on Mars without Earth in the loop.

Frequency of use for postdrive autonomous target selection

The autonomous targeting capability has been used 52 times, obtaining postdrive autonomously targeted ChemCam measurements after just over half of the drives carried out since rollout (excluding a period in February 2017 when ChemCam was unavailable because of an instrument anomaly). In each MSL planning cycle, an activity plan for the rover is developed and uplinked. Personnel schedules and orbital mechanics make it effective to produce different types of plans. Nearly half cover a single Martian sol of activities; the rest are multisol plans covering two or three sols or, occasionally, longer periods. Not every plan contains a drive, but in general, there is at most one drive per plan and so at most one opportunity for postdrive autonomous targeting.

When the plan contained a drive, AEGIS has been used 53% of the time. For single-sol plans only, this value is 36%, and AEGIS is used in 64% of multisol plans with drives. Figure 6 presents an analysis of all MSL planning cycles since AEGIS rollout, showing how often the science operations team chose to include AEGIS for postdrive autonomous targeting. The most common reason for a plan not to include it is the absence of a drive. Also shown in Fig. 6 is the fraction of cases when time limits prevented AEGIS inclusion and instances where other planning resources were the limit, including power and data volume. Communication limits between Mars and Earth restrict the amount of data that can be returned in each planning cycle, and this often means choosing which observations must be sacrificed, not because they are undesired but because the data budget is full. Inclusion of energy-intensive activities in a plan can also restrict the options for ChemCam or other observations and thus for AEGIS use.

Fig. 6 When postdrive autonomous targeting is used.

In each mission planning cycle, the science operations team may choose to use AEGIS postdrive autonomous targeting. They will not use it if the plan does not include a drive.

In a few cases, the team decided that a very short drive (less than 10 m) did not warrant new AEGIS bedrock surveys postdrive. “Other,” in the chart, refers to an initial restriction that the very first use of AEGIS after rollout should be fully downlinked and assessed before the second usage; one drive plan occurred before that assessment was complete, and AEGIS was therefore not used after that drive.

Pointing-refinement application

The autonomous pointing refinement capability has been used only twice; because it is intended to be used only when very small targets present a pointing reliability challenge (when the extra time for acquiring and processing the source image is worth taking), this reflects the rarity of such features in the rover’s recent surroundings. The initial RMI target profile was tuned for bright (whitish) veins in outcrop, which have been commonly seen in the rover’s working area and are frequently targeted, sometimes with difficulty because of their small angular size. The experience in the terrain seen since AEGIS became available is that often, when small veins are present, demonstrably related larger veins are also available, which are of sufficient size that they can be reliably targeted without AEGIS’ help.

Nonetheless, when called upon, this capability has performed very well. As shown in Fig. 3, in both cases where AEGIS was used to refine the pointing, the initial pointing missed the vein target, and AEGIS recognized the offset and corrected the pointing. In each case, this saved the observation; it would have been necessary to make a second attempt in a subsequent plan, assuming time was available before the rover’s next drive.

DISCUSSION

AEGIS has quickly become an important part of the MSL exploration process, and the postdrive autonomous targeting application in particular is regularly used to enhance the field studies the team can conduct at each location the rover visits.

Over the past 2 years, Curiosity has, for the most part, been traversing through finely stratified lacustrine mudstones and relatively fine-grained sandstones of the Murray formation, which spans nearly 200 m of vertical elevation on the lower slopes of Aeolis Mons. One of the principal scientific questions is whether the Murray formation chemistry varies over the presumably long time span in which these layers were deposited, which could reflect differences in sediments input to the ancient Gale Crater lake or changes in chemistry of the lake water. For this type of study, a large number of reasonably closely spaced bedrock observations are desired. By tuning AEGIS to autonomously target a typical bedrock profile, the MSL Science Team is able to use the postdrive periods for this high-priority, but routine, measurement. AEGIS has observed 49 bedrock targets, which is a significant addition to the bedrock targets selected by the ground crew. These observations have also freed the ground crew to select other types of targets, leaving the bedrock to be selected by AEGIS.

When AEGIS data can be downlinked in time for the next planning cycle, the MSL Science Team uses postdrive AEGIS observations to obtain knowledge of new sites before planning new observations, something that is otherwise only obtained for multisol stops. This provides science team members with important additional information for maximizing the scientific value of science blocks and allows for potential follow-up measurements with other rover instruments if requested.

Choosing the targets the science team prefers

Autonomous interpretation of images of natural scenes is a challenging task, and AEGIS’ performance in regularly parsing such scenes in new terrain that has never been seen before is key to its utility to the mission. Since the parameter update on sol 1400, the rover has driven over 2.5 km into previously unexplored areas of Mars’ surface, including crossing new geological units and environments ranging from Bagnold dunefields with few rocks to the rocky Naukluft plateau with large expanses of outcrop (both named informally by the science team). Along that traverse, 47 AEGIS-selected targets were measured by ChemCam.

Of these, only two measurements (sol 1516 and sol 1519) were on loose float rocks, rather than the preferred bedrock. On sol 1516, the postdrive field of view presented to AEGIS contained very little outcrop (as seen in Fig. 2); the sol 1519 field of view did have significant outcrop but included a float rock, which better matched the ranking parameters in the target profile.

In one case (sol 1660), the top-ranked target was outcrop, but the second-ranked target included a small amount of sand covering the outcrop. Three of the nine LIBS points fell on the sand (yielding, as it turned out, a useful sand chemistry measurement as part of a characterization of nearby sand dunes), with the remaining six points striking outcrop (achieving the goal of a geochemical measurement of it). The sol 1660 scene contains very little rock, being dominated by an expanse of sand (see Fig. 2). All other AEGIS-selected ChemCam targets since the sol 1400 parameter refinement have exclusively measured local bedrock and contributed significantly to the geochemical survey along the rover’s traverse up the lower slopes of Aeolis Mons.

AEGIS’ consistency in target selection was successfully tested during an accidental computer vision experiment. The sol 1441 drive failed to execute (for reasons unrelated to AEGIS), so AEGIS’ field of view was virtually identical to the sol 1439 run, though with very different lighting because it ran at a later time of day. The centroid of AEGIS’ top target on sol 1441 was a very slightly different part of the same rock that was selected on sol 1439, and overall, the target results were very similar on the two sols.

Regular use as a sign of a valued tool

AEGIS is not an experimental system or a concept demonstrator to be evolved into a practical implementation; it serves aboard a major planetary exploration mission as an operational tool. The goal of the system development was to provide the MSL Science Team with an operational tool to increase data return from ChemCam, provide data faster (when timing of downlink opportunities allows) to aid planning, and increase the scientific productivity of the mission. Thus, a key outcome is the degree to which the system has been adopted by the science team for use.

AEGIS competes for resources (time, data, and power) with other observations, many of which can be carried out without ground-in-the-loop targeting (such as wide-field imaging or environmental measurements). The science operations team, seeking to maximize the return from the mission’s observations, chooses which activities to conduct in each planning cycle. The fact that the team chooses to use AEGIS after most drives argues strongly that the science team places value on its capability.

The higher rate of use in multisol plans (after 64% of drives, as compared with 36% in single-sol plans) reflects the larger amount of postdrive science time in those plans—often an entire sol of the plan, owing to the need to downlink the drive data in time to plan the next cycle. AEGIS affords the opportunity to make effective use of time after the drive to obtain ChemCam measurements at the postdrive location, freeing up time in the next plan’s predrive period. The situation is further helped by the greater flexibility in managing onboard resources across several sols—data budgets and, to some extent, power consumption can be moved between sols, giving more options to fit activities into a plan, which might not fit together in a single sol. Even in single-sol planning, plans with drives include AEGIS more than a third of the time, where it typically makes use of the time after the drive, which would otherwise be unused or restricted to nontargeted observations because of planning constraints. In particular, communications schedules sometimes impose a time of day by which the drive must be completed; AEGIS can recover the time after the drive in these cases. When this scheduling constraint is not imposed, the team often prefers to push the drive later, maximizing predrive ground-targeted observation time at the cost of postdrive autonomously targeted observations.

Enhancing human-planned science campaigns with autonomous observations

The postdrive ChemCam observations made by AEGIS can be additional to those made in the next plan (once Earth has been in the loop) or can replace them, freeing up ground-targeted science time for other activities. For example, on sol 1441, the science operations team found that the rover’s workspace contained four distinct rock types; resources only permitted measuring three of them with ChemCam, but AEGIS had already collected data on the fourth material postdrive in the previous plan. Similarly, on sol 1577, AEGIS’ sol 1576 outcrop measurements were used to justify spending targeted time on an unusual float rock, which was suspected to be a meteorite. The justification was made even though the AEGIS results had not yet been transmitted to Earth, on the basis of AEGIS’ consistent performance. In both cases, AEGIS-selected observations allowed the mission to achieve a more complete study of the geological materials in the rover’s area.

Serendipitous observations

Ground-targeted observations are typically quite deliberate—the team chooses rocks with the intent to understand visibly salient materials or to test a hypothesis about the rocks in the rover’s workspace. However, occasionally unexpected differences can appear, which might not have been apparent. The sol 1612 AEGIS observation, for example, showed the highest concentration of chlorine ever measured by ChemCam, expanding the envelope of known chemical compositions in the area rocks.

Such discoveries can influence subsequent activities and observations. The sol 1371 AEGIS-selected observation detected silica enrichment a short drive from a recent drilling location where characterization of silica content had been a key goal; the next plan focused on follow-ups to survey silica content in response to this discovery. In another case, the sol 1662 observation showed unusual enrichment in phosphorus, manganese, and iron, prompting the team to use a targeted measurement to revisit that same feature.

None of these targets were visually distinct, and it is unlikely that they would have been selected by the human operators, but each led to new geochemical insights, which are still being studied by the science team.

Limitations

AEGIS is subject to the constraints of the operating mission; resources are limited, and time, power, and data volume are all precious. Fortunately, the ChemCam measurements that AEGIS enables are valuable to the science team, but they must compete in each planning cycle with other tools to study the rover’s surroundings. Numerous operational restrictions also exist, owing to safety requirements for Curiosity and ChemCam, which limit the range of targets and types of observations available on them. The initial operational rollout relies, for the NavCam case, on a single, standard field of view, which optimizes the targetable area within these constraints. New, more capable operator tools are being developed and validated to allow greater flexibility for the operations team to select the source image field of view, the number and details of follow-up observations, and the preferred target type.

AEGIS relies on computer vision, interpreting natural scenes. It has performed consistently, running with different solar illumination conditions, but shadows can be a challenge, and it will occasionally identify a rock’s outline as including part of its shadow. The targeting strategy (aiming for the centroid of targets) and the focus on light-toned, low-relief outcrop targets significantly mitigate this effect.

AEGIS’ target profiles are realized as a combination of computer vision parameters for target finding, filtering, and ranking; semantic knowledge of the appearance of rock outcrops must be expressed mathematically with these parameters. The system has shown itself adaptable to dark float rocks, light-toned outcrop, and bright veins but will naturally perform better in some types of scenes than others.

Implementation

Teams aiming to make use of AEGIS-style autonomous targeting should consider it early in the development of their mission. An understanding of the dynamics of planning, the communications cycle with the spacecraft, and the types of targets likely to be studied is all helpful in assessing the suitability and utility of such a system to the science mission. The farther from Earth (both literally and in a communications-cycle sense), the more likely that science autonomy of this type will enhance or even enable the mission.

MATERIALS AND METHODS

AEGIS framework

The process of autonomous targeting effected by AEGIS is illustrated in Fig. 7. The process begins with the acquisition of the source image—either a NavCam stereo pair or a ChemCam RMI image. The scene in this image is used for target detection in the next step. AEGIS uses a combination of computer vision techniques to identify potential rock targets for follow-up observation, described more thoroughly in (38). They importantly include an algorithm called Rockster (rock segmentation through edge regrouping) that makes use of edge-detection techniques and is described below. With targets identified, basic image analysis techniques are used to identify target features that can be used to select between targets. These include a number of features relating to pixel values (intensity and variation of intensity), the geometry of the target (shape, size, orientation, and smoothness of the perimeter), and position in space (derived from stereo data). These features are then used for target filtering, where targets with particular characteristics can be excluded from consideration (such as very small targets), and target prioritization, where targets can be ranked in importance based on these characteristics. The AEGIS target profiles are built from combinations of Rockster parameters and target filter and ranking settings.

Fig. 7 AEGIS autonomous targeting process.

Illustration of the steps taken for selecting and observing a target with AEGIS and ChemCam. The backward-pointing arrow indicates that multiple targets can be observed by ChemCam from a single AEGIS image analysis.

For any of the detected targets (generally the top-ranked targets), the system can place a visual target tracking [VTT (39)] coordinate frame with origin on the target. Follow-up observations are then acquired, pointing in the targeting frame to acquire ChemCam observations of the target. The system can be configured, at each use, to conduct as many ChemCam observations as desired (and resources allow) and can use different target filtering, target prioritization, or source images for each observation.

Rockster

Rockster (40) performs onboard image analysis to identify closed object contours, effectively reducing millions of pixels into a handful of scientifically meaningful objects that can be further analyzed and prioritized by AEGIS. Rockster was originally designed to find rocks and regions of outcrop in NavCam images; however, the same code has been applied to RMI images to successfully locate mineral veins and other small-scale features not originally envisioned.

At its core, Rockster uses gradient-based edge detection and linking steps similar to the Canny edge detector (41). Because only local information is used at this stage, closed boundary contours rarely result. The “full” version of Rockster proceeds by splitting initial contours into smaller fragments based on salient points and potential T-junctions. A gap-filling mechanism establishes new links between the end points of the resulting fragments. Background flooding from outside the image borders (or, optionally, from inside the identified contours) identifies enclosed regions. After morphological cleanup, a list of points along each boundary contour is generated.

Because of strict computing resource constraints on MER, a “reduced” version of Rockster was deployed on the Opportunity rover. The reduced version omits contour splitting at salient points and T-junctions. Hence, the gap filler is applied using only the end points of the original edge contours, which results in slightly suboptimal segmentation performance. This simplified implementation (also used on MSL) has nonetheless allowed effective target detection for AEGIS and ChemCam.

System validation and mission integration

AEGIS benefits from the previous deployment to the MER Opportunity rover (35), but the MSL mission has a number of differences, notably in the rover, its software, its flight computer, and the geological environment. Most significantly, AEGIS had to be adapted to process images from the RMI and to guide the targeting of ChemCam. ChemCam fires a powerful laser capable of converting solid rock to plasma and so poses a risk to the rover hardware if improperly aimed. The instrument is also sensitive enough to be damaged if pointed at or near the sun. Last, the LIBS technique requires a focused laser, necessitating accurate estimation of the distance to the target to achieve a measurement.

For all of these reasons, deployment of AEGIS to MSL involved careful and thorough demonstration of the system’s safety and reliability. New components were added to AEGIS to handle sun safety, prevent “collision” with the rover hardware, and compute stereo range to identified targets. The system was thoroughly tested in MSL’s software and hardware simulators [the workstation test set (WSTS) and vehicle system testbed (VSTB), respectively]. AEGIS activities were simulated in each to validate the performance of each step of the AEGIS autonomous targeting process and to check compatibility with other onboard processes. After the successful conclusion of these tests, AEGIS was approved for uplink and installation in the Curiosity rover’s flight software in September 2015. The installation and verification thereof were completed over several steps in October 2015, working around scheduled science activities to minimize the impact on the ongoing mission.

Once installed, the system had to be validated on Mars before it was released for use by the science team. The AEGIS checkouts were implemented as a seven-step process with each step allowing progressively more control of rover systems by AEGIS. Autonomous target selection was tested in three steps:

(1) Acquire NavCam stereo image, process it to find targets, place VTT targeting frame on the top three targets, and acquire a NavCam image of each target. This step confirms that AEGIS runs correctly, processes images, finds targets, and enables targeting.

(2) As in step 1, then acquire RMI-only rasters of the top two targets. This step confirms that AEGIS can command ChemCam to point at and observe a target and checks the accuracy of ChemCam pointing when commanded by AEGIS.

(3) As in step 1, then acquire a standard RMI-LIBS-RMI raster of the top target. This final step validates the full observation process and the quality of ChemCam LIBS data.

The autonomous pointing refinement capability was tested in a similar three-step process but with the use of the RMI to obtain the source image instead of the NavCams. A final, seventh checkout step conducted autonomous target selection in NavCam, acquired an RMI image of the top target, conducted autonomous pointing refinement (for bright vein-type features) using that RMI image as the source, and acquired a ChemCam raster on the target after recentering. This validated the dual-camera use in flight and was a full-up test of all AEGIS’ capabilities.

The AEGIS checkouts were conducted over a period of 80 sols, working round mission operations schedules and aiming to minimally interfere with the science mission. In particular, the ChemCam team meeting took place in November 2015, during which no new tests were conducted. Also, the mission stood down for holidays in late December 2015 and spent much of January 2016 in an ambitious, tightly scheduled campaign to study the Bagnold Dunes, the first such dune study ever conducted on another planet. The carefully planned dune science campaign left little time for engineering tests of AEGIS.

The seven AEGIS checkout phases were conducted on sols 1157, 1160, 1184, 1187, 1191, 1198, and 1237; an attempt was also made on sol 1164, but that plan was not uplinked to the rover because of a failure of the Deep Space Network transmitter that day. With the completion of the checkouts in late January 2016, and their subsequent assessment being a success, the way was paved for training the operations teams to use the system, which was released for use in May 2016.

Sun safety

ChemCam uses a telescope with a 15-cm primary mirror to focus the LIBS laser on its target. This telescope, if pointed near the sun, can focus sunlight into the instrument with the potential to cause damage. Protecting ChemCam from this risk is an important consideration in planning observations. Operators on Earth use knowledge of the sun’s position relative to the rover and its path through the sky to design observations. Targets, and the instrument’s path in slewing between them, are selected to meet the science goals while avoiding pointing the instrument near the sun’s actual position, or any position it can occupy during that plan, with margin. A special position of the focus motor (the “sun-safe” focus position) can also be used, although ChemCam is unable to make observations with the motor in that position. Onboard software, separate from AEGIS, can check and reject an unsafe pointing command but is not relied on for planning; a rejected observation is lost time on Mars, and any failure of this software check would threaten the instrument.

AEGIS must comply with the ChemCam sun safety constraints when selecting targets. Three controls are available to this end. First, the source image pointing can be chosen to avoid the sun risk locus. In practice, the very wide (45°) field of view of the NavCam means that this greatly restricts the working area. Also, because AEGIS activities occur postdrive, the precise rover orientation (and thus the relative position of the sun) is not known with certainty during planning.

To account for this, AEGIS has two ways of filtering out targets to reject nonsafe pointings. The first allows operators to define “keep-in zones” with azimuth and elevation limits. These can be used to restrict AEGIS areas outside the sun risk locus; at run time, AEGIS will filter out targets not within a keep-in zone. Alternately, AEGIS can be instructed to reject all targets above −17° in elevation (where 0° is at the ideal horizon); this provides sufficient margin to protect ChemCam from sun exposure, even at sunrise or sunset, but also sacrifices any targets that are above −17° but outside the sun risk locus. The two sun safety commands can be used together if desired.

The current template postdrive activities use a low-elevation pointing that is sun-safe under nearly all conditions and a defined set of keep-in zones, which are updated seasonally as the sun risk locus moves throughout the Martian year. For pointing refinement runs, the human operators are responsible for sun safety, as for typical ground-targeted ChemCam activities.

Collision

The MSL rover suspension system has a large range of motion, allowing the rover wheels to be in a wide variety of positions. In addition, the robotic arm has a reach of more than 2 m. It is used to place arm-mounted instruments and tools on surface targets for delivering sample to instruments and observation tray, placing instruments on calibration targets mounted on the rover body, and various other science and engineering tasks such as positioning the arm-mounted Mars Hand Lens Imager (MAHLI) camera at 40+ positions for taking a rover self-portrait mosaic. If the rover hardware is visible in the field of view of the NavCam targeting image, AEGIS will often find targets on the rover body that match the criteria for features of interest. Not only can this result in a wasted observation if rover hardware is selected for the follow-up observation, but if the follow-up AEGIS observation includes a ChemCam laser firing, it can also result in an attempt to fire the laser at the rover body, tripping a collision fault. Therefore, MSL AEGIS has the capability to query the collision model for the current position of all rover hardware at the time the AEGIS NavCam image is taken and project it into the NavCam image, masking pixels that overlap with rover hardware. AEGIS targets that fall in the masked region of the image are removed from consideration. To date, MSL AEGIS on Mars has not selected a target on the rover body.

Stereo

The MSL version of AEGIS has the additional capability to perform stereo analysis on the NavCam image. This provides the distance to the target, its three-dimensional size, and the x,y,z position in the rover’s coordinate frame. These features are used for filtering and ranking targets. For example, targets that are outside of the acceptable ChemCam LIBS firing range are filtered out, meaning that LIBS targets cannot be closer than 2.2 m from the rover or farther away than 7 m. For ChemCam follow-up measurements, the x,y,z position is used to identify the target position, and range is used to help ChemCam determine the correct focus settings. For every target it finds, AEGIS computes stereo information using the left and right NavCam images. As already described, MSL software runs on a radiation-hardened processor and has limited processing power, necessitating a number of optimizations for AEGIS stereo processing. One is to perform point stereo at only the inscribed circle center of the feature. In addition, AEGIS stereo restricts searching for stereo matches in the right-hand image to just the allowed LIBS range, thereby failing stereo correlation outside that range instead of wasting time searching a larger region of the right-hand image.

Use in mission operations

Each operations planning cycle for MSL includes “preplanning” a draft version of the next cycle’s operations plan. The AEGIS postdrive activities are often added at this early stage, after the drive, but preferably before the final “decisional” communications pass, allowing for downlink and assessment of data before choosing other ChemCam targets in the area. Placing it before the decisional pass provides the advantage of having preliminary analyses on bedrock before the team inspects the newest downlinked images at the start of the next planning cycle, at the cost of a few minutes for AEGIS to conduct the autonomous targeting in the current cycle. Post-LIBS color MastCam imaging of the target is standard for ChemCam observations; although, in principle, the MastCam can be pointed automatically by AEGIS, in practice, these images are typically acquired in the following plan.

The postdrive activity is implemented as a reusable template; a standard pointing, a target signature, and ChemCam raster parameters permit this simplified delivery, which was selected for the initial rollout strategy to allow the operations teams to become familiar with AEGIS. The standard pointing was selected to cover a region of terrain, which is within ChemCam’s LIBS range, excludes rover hardware, offers a large area of potential targets, and is reliably sun-safe (AEGIS nonetheless rechecks the safety criteria at run time). Two templates exist—one for measuring AEGIS’ top-ranked target and the other for the top two targets.

When used, AEGIS for pointing refinement is added to a given plan on the day of uplink if time and data volume allow. This planning more closely resembles a typical LIBS raster; the ChemCam team inputs the initial pointing for the RMI source image and performs their own safety checks to ensure that the observation will run as expected. Post-LIBS documentation takes place immediately after the ChemCam activity, as is standard practice, because the target location is known to within a few milliradians.

New ground tools for the planning team will allow more flexible use of AEGIS, especially in the postdrive case, where source image pointing, ChemCam raster shape, number of targets, and target profile will all be adjustable. These tools are nearing release to the operations team as of this writing and are expected to further increase the opportunities to include AEGIS in rover plans, yet again enhancing the data return from ChemCam.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/2/7/eaan4582/DC1

Data S1. Spreadsheet with all AEGIS autonomous target selection runs on Mars and results of material hit (Excel).

REFERENCES AND NOTES

Acknowledgments: This work was conducted, in part, at the Jet Propulsion Laboratory, California Institute of Technology, under a contract from NASA. Government sponsorship is acknowledged. The AEGIS team would like to thank the Mars Science Laboratory mission and the ChemCam instrument team for support and accommodation during development, deployment, and operations. W. Rapin assisted with generation of labeled RMI images. Funding: This project was funded in part by NASA’s Advanced Multi-Mission Operations System research program, by the MSL mission, and by the ChemCam instrument team. R.F. was supported by the Natural Sciences and Engineering Research Council of Canada Collaborative Research and Training Experience program for Technologies and Techniques for Earth and Space Exploration hosted by the Centre for Planetary Science and Exploration at the University of Western Ontario early in the project. J.F. thanks the Carlsberg Foundation for support. Work by O.G. was supported by CNRS and CNES, based on observations with ChemCam on MSL. Competing interests: The authors declare that they have no competing interests. Data and materials availability: All science data from the ChemCam instrument, including those gathered on AEGIS-selected targets, are available from NASA’s Planetary Data System archives.
View Abstract

Navigate This Article