Overview

Executive Order (EO) 14110, “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” directs the Secretary of the U.S. Department of Homeland Security (DHS) to develop plans for, conduct, and complete an operational pilot using artificial intelligence (AI) capabilities to aid in the detection and remediation of vulnerabilities in critical United States Government software, systems, and networks. The Cybersecurity and Infrastructure Security Agency (CISA) provided the report on the pilot for DHS to deliver to the White House on July 26, 2024.  This operational pilot also meets CISA’s final non-recurring requirement under EO 14110.

About the Pilot

From late 2023 to early 2024, CISA performed an operational pilot to examine whether current vulnerability detection software products that use AI, including large language models (LLMs), are more effective at detecting vulnerabilities than those that do not use AI. 

Pilot Scope

To learn actionable lessons within the given timeline, the CISA team scoped the pilot with the following properties:

• The evaluation task has a clear and testable definition of accuracy; 
• The AI deployment involves a service within CISA’s existing service offerings;
• Newer AI product types, such as those using LLMs, are the primary testing focus; and
• The AI products were available for use on or before December 31, 2023.

The CISA pilot team used two scenarios for testing the AI tools: security assessments of federal partner networks, and tests within a controlled environment.

Key Findings

CISA’s findings from the pilot are: 

• The best use of AI for vulnerability detection currently lies in supplementing and enhancing as opposed to replacing, existing tools;  
• In some cases, the amount of time needed for analysts to learn how to use the new capabilities is substantial and the incremental improvement gained may be negligible; and 
• In some cases, AI tools can be unpredictable in ways that are difficult to troubleshoot.

Looking Ahead

AI tools are improving constantly, and the CISA team will continue to monitor the market and test tools to ensure CISA’s vulnerability detection capabilities remain state-of-the-art.

Additional Information

Learn more about CISA’s work in artificial intelligence by visiting cisa.gov/ai.