VFRAME is a collection of open-source computer vision software tools designed specifically for human rights researchers working with large datasets of visual media.
VFRAME grew out of discussions at a 2017 Data Investigation Camp run by Tactical Technology Collective in Montenegro. Through meeting with investigative journalists, human rights researchers, and digital activists from around the world it became clear that computer vision was a much needed tool in this community, yet the solutions were either not yet technically developed, existed in disparate states, too expensive, or were only relevant to consumer applications.
VFRAME leverages recent advancements in computer vision to bring practical applications to human rights research. The project is being piloted with researchers at Mnemonic and applied to their work in analyzing large media collections from conflict zones that may contain evidence of atrocities and war crimes. Their work requires locating as many videos as possible that could be used to reconstruct the chain of events leading to violence. But one of the main challenges in this work is the massive scale of visual data. Manually reviewing millions of videos is simply not possible, especially for a small team of experts trained to recognize illegal munitions.
VFRAME aims to assist expert human rights researchers by encoding their knowledge into algorithms that scale to meet the new challenges of OSINT investigations. For example, to locate cluster munitions in videos from Syrian and Yemen, VFRAME is developing a cluster munition detection algorithm to help automate analysis of several million videos from Syria and Yemen.
Developing custom object detection algorithms brings new technical challenges. It requires locating thousands of diverse images that can be annotated and used as training data, but this is difficult or outright impossible for illegal munitions that appear infrequently and often in low resolution. To offset this, VFRAME introduces a novel approach for using mixed-reality training data comprised of 3d-rendered, 3d-printed, and original source data. This approach uses 3D modeling to recreate objects that are then randomized in 3D environments and rendered into photorealistic imagery. To overcome the overfitting that results from repeating rigid 3D objects, 3d-printed data is fabricated and photographed in real world settings. This combination of data provides a useful, and safe, alternative source of training data for dangerous objects, such as the AO-2.5RT cluster munition.
Research and development of VFRAME is or has been supported by the ProtypeFund (DE), Swedish International Development Agency (SIDA), Meedan, and NL Net. You can reach out to the director of the VFRAME project, Adam Harvey, on the secure messaging platform Keybase at keybase.io/vframeio or send an email using your intuition.
NB: Portions of this site may be out of sync with the current state of project development. More technical information is available in the documentation notes at https://github.com/vframeio.
VFRAME is developed by Adam Harvey and Jules LaPlace and a group of friends and contributors in Berlin and around the world. Many thanks to our funders, exhibition partners, and supporters.
And:
| Dates | Objective | Status and Results |
|---|---|---|
| 2021 Q4 | Optimize core image processing | - |
| 2021 Q3 | Share results of case study | - |
| 2021 Q2 | ModelZoo documentation | - |
| 2021 Q1 | Hybrid datasets, semi-automatic annotations | pending |
| 2020 Q4 | Expand Visual Recognition Library | Finalizing |
| 2020 Q3 | Develop ModelZoo and Demos | Finalizing |
| 2020 Q2 | R&D of vision models for Syrian and Yemeni archive | Complete |
| 2020 Q1 | Public release of cluster munition detector models | Complete |
| 2019 Q4 | Generation of synthetic training datasets for AO-2.5RT/M, ShOAB-0.5, BLU-63, PTAB-1M | Complete |
| 2019 Q3 | Prototype release | Complete |
| 2019 Q2 | Develop 3D models of cluster munitions, prototype Blender software | Complete |
| 2019 Q1 | Evaluation of Blender | Complete |
| 2018 Q4 | Evaluation of Unity | Complete |
| 2018 Q3 | 3D modeling, exhibition of concept prototypes at Ars Electronica EXPORT | Complete |
This site is designed to be privacy-friendly and does not use any 3rd party analytics to track visits, nor any 3rd party dependencies that compromise privacy or share data.
The site is built with Markdown and, aside from loading images on a self-hosted external server, no other external requests are made to other sites.