About > Collaborations > Tech 4 Tracing (T4T)

 Photogrammetry scan of the 9N235 submunition by VFRAME in collaboration with Tech 4 Tracing. Image license: © Adam Harvey / VFRAME.io
Photogrammetry scan of the 9N235 submunition by VFRAME in collaboration with Tech 4 Tracing. Image license: © Adam Harvey / VFRAME.io

VFRAME partners with Tech 4 Tracing to gain insight and expertise into the munitions used in conflict zones in order to create 3D models and capture sample data for developing and enhancing computer vision object detection technologies.

Tech 4 Tracing (T4T) is an international, US-based non-profit working towards building partnerships with human rights investigation communities to enhance and improve their abilities to understand and address illicit arms flows, bring perpetrators of armed violence to justice, and contribute to safer societies.

During the winter/spring of 2022 VFRAME and T4T partnered to develop a 3D model of the 9N235 submunition that has frequently appeared in the war in Ukraine. This model is being used to create synthetic training data for object detection algorithms and high-accuracy 3D printed replicas for benchmarking purposes.

 Capturing the original 9N235/210 submunition using photogrammetry. Image license and credit: © Adam Harvey / VFRAME.io.
Capturing the original 9N235/210 submunition using photogrammetry. Image license and credit: © Adam Harvey / VFRAME.io.
 The 9N235 submunition 3D models prepared for use in 3D rendering and 3D printing. Image license and credit: © Adam Harvey / VFRAME.io.
The 9N235 submunition 3D models prepared for use in 3D rendering and 3D printing. Image license and credit: © Adam Harvey / VFRAME.io.

The 3D models are used to render synthetic training datasets for developing object detection algorithms. The 3D environment can be manipulated to change the lighting conditions, camera zoom, alter the terrain, or simulate dirt and corrosion. Rendered synthetic data can be automatically annotated, saving huge amounts of time compared to conventional, manual image annotation.

 Sample 3D rendered synthetic training data image for the 9N235/210 submunition. License: © Adam Harvey / VFRAME.io 2022.
Sample 3D rendered synthetic training data image for the 9N235/210 submunition. License: © Adam Harvey / VFRAME.io 2022.
 Detection test image and score for the real 9N235/210 submunition image. License: © Adam Harvey / VFRAME.io 2022.
Detection test image and score for the real 9N235/210 submunition image. License: © Adam Harvey / VFRAME.io 2022.
 Sample 3D rendered synthetic training data image for the 9N235/210 submunition. License: © Adam Harvey / VFRAME.io 2022.
Sample 3D rendered synthetic training data image for the 9N235/210 submunition. License: © Adam Harvey / VFRAME.io 2022.
 Detection test image and score for the real 9N235/210 submunition images. License: © Adam Harvey / VFRAME.io 2022.
Detection test image and score for the real 9N235/210 submunition images. License: © Adam Harvey / VFRAME.io 2022.
Sample video showing a simple 3D rendered scene with the 9N235 submunition

After training on the 3D rendered synthetic data, the object detection algorithm needs to be tested on real world data. However, gaining real images of munitions can be difficult due to the dangerous circumstances where they appear. Or they may be scarce, with only a few dozen images available. Ideally the algorithm should be tested on thousands of diverse images.

To overcome this challenge, VFRAME produces photo-realistic replicas using 3D printing. The painted replicas are placed in staged scenes to simulate the conditions of target including weather, lighting, and terrain. These staged images are then used as benchmark data to evaluate how well the object detection algorithms work. Below are several early examples of the (in progress) 9N235 3D printed replica.

 3D printed replica of the 9N235/210 submunition. License: © Adam Harvey / VFRAME.io 2022.
3D printed replica of the 9N235/210 submunition. License: © Adam Harvey / VFRAME.io 2022.

The 3D print is painted to match the metallic properties of the original object, stencils are applied, and then the object is placed in settings to simulate the real terrain. Here the object is covered in dirt and placed in a grassy terrain.

 A 3D printed replica of the 9N235 submunition with dirt applied. License: © Adam Harvey / VFRAME.io 2022.
A 3D printed replica of the 9N235 submunition with dirt applied. License: © Adam Harvey / VFRAME.io 2022.
 A 3D printed replica (clean version) of the 9N235 submunition with partial occlusion and benchmark detection results. License: © Adam Harvey / VFRAME.io 2022.
A 3D printed replica (clean version) of the 9N235 submunition with partial occlusion and benchmark detection results. License: © Adam Harvey / VFRAME.io 2022.

Together, the 3D rendered and 3D printed data sources enable scalable development of highly specific, dangerous, and rare objects such as cluster munitions. This approach overcomes the limitation of data scarcity and could enable a workflow for developing computer vision algorithms to detect virtually any visually unique munition.

In September VFRAME will release a public version of the 9N235 submunition object detection neural network based on the methods described above. The detector will be open sourced under the MIT license. Please check back on VFRAME.io on September 1 for more details and a web demo.

 9N235/210 benchmark dataset test. License: © Adam Harvey / VFRAME.io
9N235/210 benchmark dataset test. License: © Adam Harvey / VFRAME.io
 9N235/210 submunition benchmark test. License: © Adam Harvey / VFRAME.io
9N235/210 submunition benchmark test. License: © Adam Harvey / VFRAME.io

Update: July 15

The beta detector is nearly ready and undergoing more thorough testing. Below are several videos showing the current performance:

Low difficulty: Munition is clearly visible, shown in small to large scale. License: © Adam Harvey / VFRAME.io
Low difficulty: Munition is clean and not obstructed, shown in full 360 rotation. License: © Adam Harvey / VFRAME.io
Low-medium difficulty: Munition is embedded in ground, tailfins exposed, light dirt. License: © Adam Harvey / VFRAME.io
Medium difficulty: Munition is covered in dirt, wet, and upright. License: © Adam Harvey / VFRAME.io
Medium-high difficulty: Munition is surrounded by similar looking decoys. License: © Adam Harvey / VFRAME.io
Medium-high difficulty: Munition partially embedded in ground, light dirt, with decoys. License: © Adam Harvey / VFRAME.io
High difficulty: Munition slightly less visible, wet, with light dirt, and 4 decoys. License: © Adam Harvey / VFRAME.io

Update September 1

New test videos added using 9N235/210 3D-printed replica. The replica is a 1:1 exact copy of the original.

Low difficulty: Munition underside is clearly visible but with some wet dirt on surface. Weather is sunny and object placed in shade. © Adam Harvey / VFRAME.io
Medium difficulty: Munition is horizontal and tailfins are closed inwards. Camera exposure flickers due to lens flares. © Adam Harvey / VFRAME.io
High difficulty: Munition is partially occluded by grass and partially occluded in ground. Tailfins and main tube show scattered dirt. Recorded in direct sunlight. © Adam Harvey / VFRAME.io
High difficulty: Munition is heavily occluded in grass and partially occluded in ground. Tailfins and main tube show scattered dirt. Recorded in direct sunlight. © Adam Harvey / VFRAME.io

Notes and disclaimers:

Links: