Computer Vision
for Human Rights

VFRAME (Visual Forensics and Metadata Extraction) is a collection of open source computer vision tools designed specifically for human rights investigations that rely on large video datasets.

Human rights researchers often rely on videos shared online to document war crimes, atrocities, and human rights violations. Manually reviewing these videos is expensive, does not scale, and can cause vicarious trauma. As an increasing number of videos are posted, a new approach is needed.

VFRAME is currently working with Syrian Archive, an organization dedicated to documenting war crimes and human rights violations, to develop computer vision tools to address these challenges.

Specifically VFRAME is developing tools to detect evidence of illegal munitions including A02.5-RT/M, ShOAB-0.5, PTAB-1M; filter graphic content to reduce vicarious trauma; efficiently search for related visual media in large datasets (over 1M videos); and a web annotation platform to construct custom datasets for training computer vision models.

Throughout development we will document the process and release a report along with open-sourced code in Fall 2018.

Research in Development

  • Prototype #1: Cluster munition detector proof-of-concept
  • Prototype #2: Fast scene-summarization using feature vectors and perceptual hashing (July 2018)
  • Developing Visual Taxonomies for hierarchical multiclass object detection (July 2018)
  • Object detector for illegal munitions V2 (July 2018)

Below are previews of the current tools in development.

Prototype #1: Cluster Munition Detector

This early prototype demonstrates an object detection algorithm trained to locate cluster munitions. Created October 2017 using the YoloV2 framework. View on Vimeo

Object detection is the ability to classify and localize an object in a video frame or image. Object detection algorithms can be trained to detect custom objects, such as the AO-2.5RT cluster munition seen in these videos.

The AO-2.5RT/RTM is a specific type of cluster munition that appears frequently in videos from the Syrian conflict. The AO-2.5RT/RTM is banned by the Convention on Cluster Munitions (CCM), which "prohibits the use, transfer, and stockpiling of cluster bombs, a type of explosive weapon which scatters submunitions ('bomblets') over an area."

Researchers at Syrian Archive have already identified evidence of this munition in dozens of videos. But manually reviewing videos does not scale. Currently there are over 1.500.000 videos waiting to be reviewed. More videos contaning evidence of the AO2.5RT/RTM may still exist in these unreviewed videos.

Using computer vision to automatically detect cluster munitions will significantly accelerate researcher's work to locate evidence of cluster munitions in this large dataset.

Prototype #1b: Hierarchical visual classification of the AO2.5-RT cluster munition. Created June 2018 using YoloV3. View on Vimeo
Prototype #1c: Hierarchical visual classification of the AO2.5-RT cluster munition. Created June 2018 using YoloV3. View on Vimeo

This project is under active development. Code will be released in October and demos in November 2018.

 

VFRAME is developed by Adam Harvey and contributors in Berlin. VFRAME is currently funded by a grant from PrototypeFund.de and Bundesministerium fur Bildung und Forschung.

 

News and Mentions