Research ArticleCOMPUTER VISION

Learning sensorimotor control with neuromorphic sensors: Toward hyperdimensional active perception

See allHide authors and affiliations

Science Robotics  15 May 2019:
Vol. 4, Issue 30, eaaw6736
DOI: 10.1126/scirobotics.aaw6736
  • Fig. 1 Tension minimization.

    An example of how tension is minimized in a graph represented the co-occurrences of vertices. As the total energy in the system decreases with each iterations, the positions of the HBVs associated with each vertex are in geometrically “better” locations, relative to each other. When proximal force is turned off, the system is enabled to reach a singularity state, and thus, the tension will naturally reach 0. However, when the proximal force is present, it is unlikely that a state that fully minimizes the oscillations will be found, and the minimization will converge at a nonzero solution.

  • Fig. 2 Intensity minimization.

    A visualization of intensity minimization. We start with random vectors for four intensities ranging from 0 to 3 (A). A complete graph is formed, connecting each vertex to each other, with decreasing weight in proportion to the intensity difference. When minimization is performed, the result forms a line or semicircle of appropriate distances (B).

  • Fig. 3 Hamming distance visualization of intensities.

    Visualization of the Hamming distance between intensity values in the range of 0 to 25. Note the increasing distances away from the diagonal, as one would expect. For DVS or images, a full 256-intensity range can be easily developed with the same properties.

  • Fig. 4 Moving pixels spatially.

    A pixel p is relocated from the origin position to row 2 and column 4 by performing a permutation R2C4p.

  • Fig. 5 Composing image encodings.

    A simple example of how an image encoding can be manipulated. The reference pixel can be arbitrarily moved through permutations as shown in (A) by permutating the current encoding appropriately [in the case of (B), this is a permutation of R−8]. A new image (C), aligned on a similar reference pixel (the origin here); the two images can be composed to construct an arbitrary image (D).

  • Fig. 6 An example of DVS data visualization.

    On the left, a time image is shown, in which green is the average timestamp, and the positive/negative per pixel event counts are shown in red/blue. On the right, the corresponding gray-scale image of the scene is shown. Note the motion blur on the classical frame.

  • Fig. 7 Hamming distance decay in data records.

    The decay in nearest neighbor matches of Hamming distance as the number of time images in a memory increases. As more frames of motion are put into the encoding, more noise is introduced, making the deviation caused by a match less and less statistically significant. At 700 frames, we are still about 3 to 4 SDs of likelihood away from random vectors.

  • Fig. 8 Results for HBVs on outdoor day 1 in the MVSEC dataset.

    Qualitative results on the outdoor day 1 subset of the MVSEC dataset. Results are split into the angular and linear speed inference. The probability plots show the likelihood of each velocity class being correct across all inferences, with a light-green color indicating the higher probabilities. One can see that the light green forms the path of predictions across the probability field.

  • Table 1 Quantitative results for the MVSEC dataset across the five subsets.

    The table gives the number of frames in each subset, length in seconds, the size of the angular and linear bins used for HBVs, number of vectors in the rotation, X and Z, and the average endpoint error (AEE), average relative pose error (ARPE), and the average relative rotational error (ARRE).

    MVSEC subsetOutdoor day 1Outdoor day 2Outdoor night 1Outdoor night 2Outdoor night 3
    Frames513412196513354975429
    Length (s)128.325304.875128.3137.4135.7
    Ang. bin (rad/s)0.020.020.020.020.02
    Lin. bin (m/s)0.080.080.080.080.08
    Rotation (clusters)104101407487
    X (left-right, clusters)4744244033
    Z (front-back, clusters)119311244251228
    AEE0.8101.030.9331.160.940
    ARPE0.1220.2250.2430.09480.0831
    ARRE0.09940.1080.06320.1160.121

Supplementary Materials

  • robotics.sciencemag.org/cgi/content/full/4/30/eaaw6736/DC1

    Text S1. MVSEC experimental details.

    Fig. S1. Theoretical likelihood of a consensus term to be 1.

    Fig. S2. Results for HBVs on outdoor day 2 in the MVSEC dataset.

    Fig. S3. Results for HBVs on outdoor night 1 in the MVSEC dataset.

    Fig. S4. Results for HBVs on outdoor night 2 in the MVSEC dataset.

    Fig. S5. Results for HBVs on outdoor night 3 in the MVSEC dataset.

    Fig. S6. Drone used in dataset collection.

    Fig. S7. Memorization pipeline.

    Fig. S8. Memory retrieval pipeline.

    Movie S1. Experimental drone and dataset.

    Movie S2. HBV representations for intensities.

    Movie S3. Encoding and memory binding.

    Movie S4. Tension relaxation.

    Movie S5. Outdoor day 1.

    Movie S6. Outdoor day 2.

    Movie S7. Outdoor night 1.

    Movie S8. Outdoor night 2.

    Movie S9. Outdoor night 3.

  • Supplementary Materials

    The PDF file includes:

    • Text S1. MVSEC experimental details.
    • Fig. S1. Theoretical likelihood of a consensus term to be 1.
    • Fig. S2. Results for HBVs on outdoor day 2 in the MVSEC dataset.
    • Fig. S3. Results for HBVs on outdoor night 1 in the MVSEC dataset.
    • Fig. S4. Results for HBVs on outdoor night 2 in the MVSEC dataset.
    • Fig. S5. Results for HBVs on outdoor night 3 in the MVSEC dataset.
    • Fig. S6. Drone used in dataset collection.
    • Fig. S7. Memorization pipeline.
    • Fig. S8. Memory retrieval pipeline.
    • Legends for movies S1 to S9

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mp4 format). Experimental drone and dataset.
    • Movie S2 (.mp4 format). HBV representations for intensities.
    • Movie S3 (.mp4 format). Encoding and memory binding.
    • Movie S4 (.mp4 format). Tension relaxation.
    • Movie S5 (.mp4 format). Outdoor day 1.
    • Movie S6 (.mp4 format). Outdoor day 2.
    • Movie S7 (.mp4 format). Outdoor night 1.
    • Movie S8 (.mp4 format). Outdoor night 2.
    • Movie S9 (.mp4 format). Outdoor night 3.

    Files in this Data Supplement:

Stay Connected to Science Robotics

Navigate This Article