Revolutionizing Data Ingestion and Curation for Intelligence Analysis

ABSTRACT

The ability to ingest properly curated data into analysis tools is of fundamental importance to exploit information from massive amounts of data sources. When analyzed properly, the data can provide insights into relationships between entities, patterns of life, and even weak signals to suggest threat activities for intelligence analysis. The more complex the data sources are, the more difficult and time-consuming ingestion becomes, requiring dedicated data engineering teams whenever a data source with new schemas or structures is added to the ingestion pipeline. The United States Irregular Warfare Technical Support Directorate (IWTSD) and Singapore’s Defence Science and Technology Agency (DSTA), the Seekers for this Challenge, are seeking a novel solution for data ingestion and curation which can adapt its method of data ingestion and remain agnostic to data schema, types, and formats.  

This is a Reduction-to-Practice Challenge that requires written documentation, output from the proposed algorithm, and submission of source code and documentation for the proposed algorithm upon Seeker’s request.

 

OVERVIEW

Intelligence analysts often must grapple with a wide range of data sources, types, structures, and formats to painstakingly prepare and curate the data before data ingestion for deeper analysis. Data ingestion efforts often rely on highly customized and individually scripted data connectors to bring data onboard which can be time-consuming. Often a dedicated data engineering team is required to process new data sources with new schemas or structures in order to add them to the ingestion pipeline. To enable analysts to efficiently and effectively produce actionable intelligence, the Seekers for this Challenge are looking for an innovative solution to automate data ingestion and remain agnostic to the various types, formats, and schema of data which can comprise text, images, videos, Excel, and CSV files.

This Challenge will be conducted in two phases. Phase 1 (from 13 Dec 2022 to 12 Feb 2023) is open to all Solvers who meet the stated eligibility requirements (see below) while Phase 2 (from 18 Feb – 22 Mar 2023) participation will be limited to the Solvers with the top three solutions from Phase 1 to evaluate and refine their unique solutions with more challenging datasets.

The Solvers with the top three solutions to Phase 1 that meet the technical requirements of the Challenge will each receive awards of $8,000 to offset the cost of participation and travel for Phase 2, which culminates with a final in-person presentation event in Singapore on 22 March 2023. Phase 2 awards will be 1^st Place - $20,000; 2^nd Place - $10,000, and 3^rd Place - $6,000. Solvers, or at least one member of a Solver team, must be present at the final event to be eligible for any award (for both Phase 1 or Phase 2). In addition, the Solver team representative will be extended a complimentary ticket to attend the Singapore Defence Technology Summit to be held from 22 – 24 March 2023.

The Challenge award is contingent upon evaluation and validation of the submitted Solutions by the Seekers.

To receive an award, the Solvers will not have to transfer their exclusive IP rights to the Seeker. Instead, Solvers will grant to the Seeker a non-exclusive license to practice their solutions upon receipt of a Phase 2 award. 

Submissions to Phase 1 of this Challenge must be received by 11:59 PM (US Eastern Time) on 12 Feb 2023.
Late submissions will not be considered.

 

ELIGIBILITY

Participants must certify they do not have identical or essentially equivalent work funded by another U.S. or Singaporean government agency.

U.S. or Singapore government employees acting within the scope of their employment are not eligible to participate. A U.S. or Singapore government employee acting outside the scope of his or her employment should consult his or her ethics official before participating in the Challenge.

Employees of ST Engineering are not eligible to participate in this Challenge.

Eligibility is subject to verification by U.S. and Singapore government before cash prizes are awarded.

Additional Criteria by U.S. Government
Participants (residents or entities) who are designated by the United States Treasury’s Office of Foreign Assets Control, or are citizens of Russia, China, Iran, North Korea, Syria, Cuba, and Venezuela, are not eligible to participate in the Challenge. Participants who are listed, or become listed, on the Excluded Parties List found on www.sam.gov, have any active exclusions, or are otherwise unable to be deemed responsible in accordance with FAR 9.104-1 based on information available in FAPIIS, are not eligible to receive any cash prize in the Challenge.

 

ABOUT THE SEEKERS

The Irregular Warfare Technical Support Directorate (IWTSD) provides a forum for interagency and international users to discuss mission requirements for Irregular Warfare, prioritize those requirements, fund and manage solutions, and deliver capabilities. The IWTSD accomplishes these objectives through rapid prototyping of novel solutions developed and field-tested before the traditional acquisition systems are fully engaged. This low-risk approach encourages interdepartmental and interagency collaboration, thereby reducing duplication, eliminating capability gaps, and stretching development dollars.

The Defence Science and Technology Agency (DSTA) is a top-notch technology organisation that drives innovation and delivers state-of-the-art capabilities to make the Singapore Armed Forces a formidable fighting force. Harnessing and exploiting science and technology, engineers and IT professionals at DSTA leverage multidisciplinary expertise to equip soldiers with advanced systems to defend Singapore. DSTA also contributes its technological expertise to support national-level developments. To achieve its mission, DSTA excels in systems engineering, digitalised platforms, cyber, software development and more.

In Partnership with

ST Engineering is a global technology, defence and engineering group with offices across Asia, Europe, the Middle East and the U.S., serving customers in more than 100 countries. The Group uses technology and innovation to solve real-world problems and to enable a more secure and sustainable world/planet through its diverse portfolio of businesses across the aerospace, smart city, defence and public security segments. Headquartered in Singapore, ST Engineering has more than 23,000 employees worldwide with two-thirds in engineering and technology roles.

 

What is the Wazoku Crowd?

Imagine a world where any problem you have can be looked at by millions of expert, motivated eyes. The Wazoku Crowd makes this possible. With twenty years of experience in crowdsourcing, the Wazoku Crowd is the gold standard for open talent and external innovation solutions. Commercial, governmental and humanitarian organizations engage with the Wazoku Crowd to solve problems that can impact humankind in areas ranging from the environment to medical advancements.

What is an RTP Challenge?

A Wazoku Crowd RTP (Reduction to Practice) Challenge is a prototype that proves an idea, and is similar to a Wazoku Crowd Theoretical Challenge in its high level of detail. However, an RTP requires the Solver to submit a validated solution, either in the form of original data or a physical sample. Also, the Seeker is allowed to test the proposed solution. For details about treatment of Intellectual Property (IP) rights, please see the Challenge Specific Agreement.