How a satellite learned to classify cloudy image tiles

A satellite does not need to send every image to Earth before software can begin interpreting it. In a July 17, 2023 preprint, researchers deployed a lightweight model called RaVAEn on D-Orbit’s ION SCV004 and demonstrated a second step aboard the spacecraft: training a small image-processing task using the model’s existing representations.

The result is a focused computational demonstration, not a complete autonomous mission. Its importance lies in showing how a satellite can reuse a visual representation and adapt a downstream task locally. To understand what actually learned in orbit, it helps to separate the original model from the task trained during the experiment.

A reusable visual shorthand comes first

RaVAEn is based on a variational autoencoder, or VAE. In simple terms, a VAE is trained to turn an image into a compact set of numbers and then reconstruct the original image from those numbers. The numerical middle layer is intended to retain useful information about the image while using a more manageable representation for later tasks.

The representation used in this study came from a VAE trained on Sentinel-2 L1C imagery from the WorldFloods dataset. That initial representation training is different from what the researchers demonstrated on the satellite. They did not train the full VAE from scratch in orbit. Instead, the spacecraft used the pretrained model to encode image tiles, then trained a downstream classifier using the resulting latent representations.

The demonstration task separated tiles into cloudy and non-cloudy categories. This division made it possible to test whether a small new task could be learned from the model’s compact numerical output without repeating the much larger representation-learning process.

Two timings measure different parts of the job

The preprint reports an encoding time of 0.110 seconds for a tile covering 4.8 by 4.8 kilometres. Encoding is the step in which the image passes through the existing model and becomes a latent representation. It is not the time needed to train the classifier, nor is it a complete end-to-end mission response time.

The study also reports that one training epoch averaged 0.091 seconds with a batch size of 256. An epoch is one pass through the training data under the stated experiment. This number describes a unit of the downstream learning process; it cannot be added directly to the encoding time to produce a total response time for an operational system.

The researchers compared running RaVAEn on the satellite’s quad-core x86 64-bit CPU with using an available Intel Movidius Myriad X vision processing unit. The evidence establishes the onboard setup and the measured operations, but the two figures should remain tied to their respective jobs rather than being treated as a single speed claim.

The cloud test shows a useful but limited capability

For the demonstration, the researchers selected 1,305 cloudy and non-cloudy image tiles for training. They evaluated the classifier on the non-overlapping remainder of the data. At a confidence threshold of 0.5, the reported precision was 0.967 and recall was 0.946, with an F1 score of 0.956. The area under the precision-recall curve, which summarizes performance across thresholds, was 0.979.

In everyday terms, precision asks how often the tiles labelled as cloudy were actually cloudy. Recall asks how much of the cloudy-tile set the classifier successfully identified. The reported values indicate strong performance on this particular held-out evaluation for the task as defined.

That task was classification at the level of whole image tiles. It did not outline the exact shape of clouds within each tile pixel by pixel. A tile-level answer can still support a rough estimate of how much of a scene may be cloudy, but it is a different capability from semantic segmentation, which would assign a cloud label across individual pixels.

The authors present the cloud classifier mainly as a demonstration of the training process and its timing. The scores therefore show that the downstream task worked on the evaluation data; they do not by themselves establish a complete onboard disaster-response workflow, sustained mission performance or a general-purpose system for interpreting every kind of satellite image.

What onboard learning could change for satellite missions

The practical possibility is local adaptation. A spacecraft could use a reusable image representation and train a narrower processing task after deployment, rather than depending entirely on model updates sent from a ground station. That could matter when communications are constrained and the mission needs its image-processing behavior to respond to data encountered in operation.

For that prospect to become useful, the learned task would still need to remain dependable with mission data, available computing resources and the spacecraft’s operating constraints. The ION SCV004 experiment establishes an onboard deployment, fast measured operations and a small training demonstration. It presents the ingredients for more adaptive satellite processing, while leaving mission-wide utility and operational reliability as questions for further validation.

Key Sources

• Primary source: Fast model inference and training on-board of Satellites
  arXiv · Preprint · 2023-07-17