Research: 3D Printed Training Data

 3D-printed AO-2.5RT parts used to fabricate inert cluster munition replica. The replica is painted, photographed, then annotated for use in computer vision training datasets. Photo: ©Adam Harvey / VFRAME.io 2021. Reuse permitted with attribution. Download hi-res
3D-printed AO-2.5RT parts used to fabricate inert cluster munition replica. The replica is painted, photographed, then annotated for use in computer vision training datasets.
Photo: ©Adam Harvey / VFRAME.io 2021. Reuse permitted with attribution. Download hi-res

High-quality training data is essential to developing accurate object detection algorithms. But most publicly available datasets are too generic for applications to human rights investigation, and recording videos of real munitions in conflict zones can put lives at risk. To offset the challenge of sourcing visual training data from conflict zones, VFRAME creates 3D-printed replicas to enrich training datasets.

3D-printed replicas are safe, inert, proxy objects that can be photographed and annotated in various environments for use in computer vision image training datasets. The examples on this page show several inert cluster munition replicas that were fabricated in 2019-2020 using high-accuracy SLA resin 3D-printing. The prints are UV-cured, sanded, painted, and then photographed in environments similar to the target domain (e.g. Syria or Yemen). The photos are then manually annotated using CVAT and fed into a object detection training pipeline using YOLOV5. Prior to training with the 3D printed data, the photos are degraded and recompressed to mimic the target domain media quality using various video codecs.

Building 3D-printed training data replicas can be time-consuming. Compared to relying on existing online sources, it requires significantly more initial labor and research. However, online sources of cluster munition training data are often sparse, incomplete, biased, or low-quality. Developing 3D-printed replicas can reduce reliance on unstable training data sources and allows the developer to control the means of producing more training data and embedding useful biases into the training dataset.

VFRAME is testing this methodology in collaboration with Mnemonic, an organization dedicated to preserving evidence of atrocities and war-crimes in Syria, Yemen, and Sudan. Together with Mnemonic, VFRAME will introduce this approach to analyze and organize their datasets with millions of videos from Syrian and Yemen to help locate evidence of illegal cluster munitions.

The images below show 3D-printed parts of the AO-2.5RT/M, BLU-63, and PTAB-1M. Additional 3D-prints are in progress.

 3D-printed pieces of a BLU-63 cluster munition replica. The replica is painted then photographed and annotated for use in computer vision training datasets. Photo: ©Adam Harvey / VFRAME.io 2020. Reuse permitted with attribution. Download hi-res
3D-printed pieces of a BLU-63 cluster munition replica. The replica is painted then photographed and annotated for use in computer vision training datasets.
Photo: ©Adam Harvey / VFRAME.io 2020. Reuse permitted with attribution. Download hi-res
 3D-printed pieces of the BLU63 and PTAB-1M cluster munition replica. Made in Berlin with FormLabs SLA resin printer to create highly accurate facsimile. Photo: © Adam Harvey / VFRAME.io. Download hi-res
3D-printed pieces of the BLU63 and PTAB-1M cluster munition replica. Made in Berlin with FormLabs SLA resin printer to create highly accurate facsimile.
Photo: © Adam Harvey / VFRAME.io. Download hi-res

The 3D-prints are hollow, inert, lightweight, and entirely comprised of plastic. Often several attempts are needed to determine the most durable construction because small pieces or decorative elements can become brittle if printed incorrectly. Once the design is proven, multiple objects can be reprinted fairly quickly. SLA resin is the preferred printing method to obtain highly accurate details and reduce unrealistic banding that is common with fused deposition modeling (FDM) printers. After sanding out the rafting artifacts, the next step in producing a data replica would be to determine a suitable paint and then burnish, degrade, or slightly damage the surface appearance to match the real world appearances seen in conflict zone documentation videos.

One of the advantages to the more laborious process of developing 3D-printed data, compared to 3D-rendered data, is how quickly new scenes can be staged and how easily debris can be added to modify the object's appearance. Although adding procedural dirt texture is fairly simple in 3D modeling environments, modeling realistic obfuscations can be time consuming. In real world environments, this is as simple as throwing sand or mud over the object. Below is an example of the AO-2.5RT cluster munition mimicking a typical documentation photo from Syria. Often the AO-2.5RT appears covered in dirt and partially stuck in the ground because it failed to detonate. For this reason, the unexploded device can be dangerous and should not handled without appropriate UXO training. The 3D prints can be safely handled and pose no risk to the handler.

 3D-printed AO-2.5RT/M cluster munition replica. Made in Berlin using FormLabs SLA resin printer to create highly accurate facsimile. Photo: © Adam Harvey / VFRAME.io. Download hi-res
3D-printed AO-2.5RT/M cluster munition replica. Made in Berlin using FormLabs SLA resin printer to create highly accurate facsimile.
Photo: © Adam Harvey / VFRAME.io. Download hi-res
 Painted, photographed, annotated AO-2.5RT/M cluster munition replica 3D-print. This image is used for training a cluster munition object detection algorithm. Photo: © Adam Harvey / VFRAME.io. Download hi-res
Painted, photographed, annotated AO-2.5RT/M cluster munition replica 3D-print. This image is used for training a cluster munition object detection algorithm.
Photo: © Adam Harvey / VFRAME.io. Download hi-res
 Painted, photographed, annotated AO-2.5RT/M cluster munition replica 3D-print. This image is used for training a cluster munition object detection algorithm. Photo: © Adam Harvey / VFRAME.io. Download hi-res
Painted, photographed, annotated AO-2.5RT/M cluster munition replica 3D-print. This image is used for training a cluster munition object detection algorithm.
Photo: © Adam Harvey / VFRAME.io. Download hi-res

Future updates to this page will provide more details about the results of training with various permutations of 3D-printed, 3D-rendered, and original videos. Follow the VFRAME project on https://github.com/vframeio for code updates and documentation. This research is a work in progress. For information about how to get involved with this project, please contact researchers at Mnemonic.

 The first public display of the 3D-printed cluster munition replicas for an exhibition at Ars Electronica in Linz in 2019. These same replicas are used for generating training data. Photo: © Adam Harvey / VFRAME.io.
The first public display of the 3D-printed cluster munition replicas for an exhibition at Ars Electronica in Linz in 2019. These same replicas are used for generating training data. Photo: © Adam Harvey / VFRAME.io.

Acknowledgments

3D replica prints made with FormLabs Form 2 and Form 3 SLA resin printer in Berlin. Thanks to Marcelo Coelho, Jory, Luke, and Ramaine.

Disclaimer

Content on this page is approved for public disclosure. Actual system capabilities, results, and benchmarks may differ from those described here.