Foundation AI Logo

AI in Cybersecurity Policy and Practice Insights — July

Alie Fordyce

AI is rapidly transforming cybersecurity—not only by enabling attackers to launch increasingly sophisticated and scalable threats, but also by reshaping how security defenses operate. As organizations confront these evolving challenges, open-source innovation and the strategic use of AI are becoming increasingly important for staying ahead of novel threats and automating defenses where possible. Recent policy initiatives further highlight the role of open source in fostering transparency, collaboration, and stronger security in AI development.

In this installment of our monthly series we will keep you up-to-date on AI in cybersecurity policy and practice insights to help inform a modern approach to cybersecurity that leverages AI. Regularly sourcing cybersecurity policy and threat information from government agencies and reputable thought leaders reveals the following key trends:

  • Large Language Models are now capable of functional cyber attacks: AI systems are now autonomously discovering and exploiting vulnerabilities—and even outperforming professional penetration testers and leading bug bounty hunters—signaling the rise of AI-driven cyber threats and defenses.
    • Claude Opus 4 (among other large language models) is being used to discover and exploit software, opening the way for real AI malware– including actually solving penetration testing.
    • The leading hacker on HackerOne is now an AI agent called XBOW, which has surpassed all human testers in performance and marks a significant shift toward AI-driven security research and automated vulnerability discovery.
  • Ongoing investments to discover and remediate vulnerabilities in code: Recent research highlights both the promise of AI-driven tools in finding and fixing code vulnerabilities. A couple papers we found interesting from Joshua Saxe’s “AI Security Notes” newsletter recently include:
    • Generation‑time self‑patching; SGCode (NJIT + UIUC) inserts static analyzers into the generation loop so GPT‑4 corrects its own output before it ever reaches the developer, eliminating many OWASP‑top‑10 issues with negligible latency.
    • Retrieval‑augmented fixes; SOSecure (CMU) shows that fetching relevant Stack Overflow discussions after an LLM produces code raises GPT‑4’s fix‑rate on LLMSecEval from 56 % to 91 %, as community knowledge gives the model concrete examples of how others hardened similar snippets.
    • Berkeley’s new CyberGym benchmark shows how hard AI for code security in the real world will be; the best LLM+scaffolding combo reproduced 11.9 % of real‑world bugs scattered across 188 open‑source projects, struggling most with logic errors that hinge on project‑specific assumptions.
    • SecRepoBench (April 2025), a 318-task benchmark built from 27 real C/C++ repos; testing 19 leading LLMs showed they “struggle with generating correct and secure code” across 15 CWE categories.
    • Georgetown CSET issued the brief “Cybersecurity Risks of AI-Generated Code” (Nov 2024) containing a detailed study of general weaknesses in LLM code and warning of a feedback loop where insecure AI code contaminates future training data.
  • AI Action Plan supports importance of open source in playing a crucial role in the security of AI development: Openness is essential in cybersecurity, enabling strong data privacy and security, giving practitioners full control over deployments and harnessing the power of community-driven innovation. The White House recently released an AI Action Plan, supporting the importance of open source innovation.
    • The AI Action Plan encourages open source AI because it facilitates access to computing power for researchers and startups, and promotes the adoption of open-source and open-weight AI models. The plan states open source innovation is a means to enhance privacy and security of an organization’s sensitive data.

Bottom Line: With AI now outpacing human hackers and automating both attacks and defenses, cybersecurity is rapidly being reshaped—making smarter security processes and open collaboration essential to stay ahead of new threats.