🔐🤖 A curated list of AI/LLM security tools, frameworks, guides, papers, and training — focused on open-source and community resources.

---

The awesome-ai-security repository is a community-driven collection of AI Security, LLM Security, and Prompt Injection tools and resources. The focus is on open-source tools and resources that benefit the community.

This repository covers:

• Large Language Model (LLM) security testing and vulnerability assessment
• Prompt injection attacks and defenses
• AI red teaming and adversarial testing
• Jailbreak detection and prevention
• Model poisoning and extraction attacks
• Hallucination detection and prevention
• AI application security best practices
• MLSecOps and LLMOps security
• Model Context Protocol (MCP) security
• AI supply chain security

📌 Please read the Contributions section before opening a pull request.

---

Name	Author	Description
garak	NVIDIA	LLM vulnerability scanner – tests 120+ categories (hallucination, data leakage, prompt injection, misinformation, toxicity, jailbreaks).
PyRIT	Microsoft	Python Risk Identification Tool for GenAI; adversarial testing automation with multi-turn orchestration.
promptmap	utkusen	Automated prompt injection scanner with white-box testing.
aiapwn	karimhabush	Automatic prompt injection testing with tailored payload generation.
FuzzyAI	CyberArk	LLM fuzzing framework to identify jailbreaks and vulns.
LLMFuzzer	mnns	Fuzzing framework for LLM API integrations.
promptfoo	promptfoo	Adaptive red teaming for LLM agents with multi-turn attacks (PAIR, tree-of-attacks, crescendo) that probe tool use, RAG, and agentic workflows. Used by 250K+ developers; featured in OpenAI and Anthropic developer education.
LLM Warden	jackhhao	Simple jailbreak detection (Hugging Face model).
Vigil	deadbits	Modular scanners (vectors, YARA, transformers) via lib & REST API.
picklescan	mmaitre314	Detects malicious code in Python pickle model files.
ModelScan	Protect AI	Multi-format ML model file scanner (pickle, SavedModel, etc.).
Open-Prompt-Injection	Yupei Liu et al.	Toolkit/benchmark for prompt injection attacks/defenses.
ARTKIT	BCG-X	Open-source framework for automated LLM red-teaming with multi-turn attacker-target interactions.
Giskard	Giskard AI	Advanced automated red-teaming platform with 50+ specialized probes and adaptive attack engine.
Mindgard	Mindgard	DAST-AI platform for automated red teaming across the AI lifecycle with artifact scanning.
CodeGate	Stacklok	Security proxy for LLMs and IDEs that filters input/output to prevent API key leakage and insecure code.
AIJack	Koukyosyumei	Open-source simulator for modeling security and privacy threats targeting ML systems.
Strix	usestrix	"AI hacker" agents for CLI & CI/CD with automated security testing.

---

Name	Author	Description
PayloadsAllTheThings – Prompt Injection	swisskyrepo	Prompt injection payloads and bypasses.
PIPE – Prompt Injection Primer	jthack	Attack scenarios and payloads for engineers.
Basic-ML-prompt-injections	Zierax	Educational payloads.
OWASP LLM Prompt Injection Prevention	OWASP	Prevention cheat sheet and best practices.
NeuralTrust AI Guide	NeuralTrust	Comprehensive guide to implementing prompt injection detection with real-time alerting.

---

Name	Author	Description
Adversarial Robustness Toolbox (ART)	IBM / LF AI	Defenses against evasion, poisoning, extraction, inference attacks with 39 attack modules and 29 defense modules.
HEART	IBM	Hardened ART extension for T&E workflows.
Rebuff	Protect AI	Self-hardening prompt injection detector (multi-layer).
PurpleLlama	Meta	Llama Guard, CyberSecEval, and more.
HarmBench	Center for AI Safety	Standardized evaluation framework for automated red teaming with 18 methods comparison.
Splx AI	Splx AI	Commercial platform for multi-modal AI red teaming with CI/CD integration.
Lasso MCP Gateway	Lasso Security	Open-source MCP Gateway for Model Context Protocol security testing.

---

Name	Author	Description
AI Goat	dhammon	Local LLM CTF challenges; no fees/sign-ups.
Gandalf	Lakera	Prompt injection game with difficulty levels (world's largest red team experiment).
LLM Security CTF	TrustAI-laboratory	Free web-based vulnerable LLM CTFs.
DamnVulnerableLLMProject	harishsg993010	DV LLM app for training/education.

---

Organization	Name	Description
SANS	SEC545: GenAI & LLM AppSec	Hands-on GenAI security.
SANS	SEC495: Building & Securing RAG	RAG security training.
SANS	SEC411: AI Security Principles	Fundamentals with Docker labs.
SANS	SANS AI Cybersecurity Summit 2025	Hands-on workshops and live demos for AI/ML integration in cybersecurity (Denver, March 31 – April 7, 2025).
AppSecEngineer	AI Combat & Construct	Attack/defend AI apps.
Practical DevSecOps	CAISP	60 days of labs; MITRE ATLAS defenses.
HackAPrompt	HackAPrompt 1.0	Prompt hacking competition.
HackAPrompt	HackAPrompt 2.0	Large-scale red-teaming hackathon.
AI Village	DEF CON AI CTF	Annual LLM security CTF.

---

Author(s)	Publisher	Name	Description
Various	Springer	Large Language Models in Cybersecurity: Threats, Exposure and Mitigation	Open-access guide (2024).
Various	Springer	Generative AI Security: Theories and Practices	GenAI impacts across security (2024).
Various	Springer	AI-Driven Cybersecurity and Threat Intelligence	AI x security (2024).
Steve Wilson	O'Reilly	Developer's Playbook for LLM Security	Practical LLM AppSec (2024).

---

Name	Author	Description
OWASP LLM Prompt Injection Prevention	OWASP	Prevention best practices.
LangChain Security Policy	LangChain	"Four Perimeters" and app hardening.
CISA AI Security Best Practices	CISA	AI system security guidance.
NVIDIA AI Red Team Practical Advice	NVIDIA	Key findings from AIRT assessments on securing AI-powered applications.
Salesforce Prompt Injection Detection Guide	Salesforce	Building trusted AI systems against prompt injection threats.

---

Name	Org	Description
OWASP Top 10 for LLM Apps (2025)	OWASP	LLM01–LLM10 risks including new entries: System Prompt Leakage (LLM07), Vector and Embedding Weaknesses (LLM08), Misinformation (LLM09), Unbounded Consumption (LLM10).
NIST AI RMF + GenAI Profile	NIST	Govern, Map, Measure, Manage.
MITRE ATLAS	MITRE	AI adversary TTPs (modeled after ATT&CK).
CISA AI Guidelines	CISA	Joint guidance for AI/ML systems.
OWASP Top 10 2025	OWASP	Updated to include A03: Software Supply Chain Failures and A10: Mishandling of Exceptional Conditions.

---

Name	Author	Description
NeMo Guardrails	NVIDIA	Programmable input/output/dialog/retrieval/execution controls.
LLM Guard	Protect AI	Runtime scanning, PII redaction, content filtering.
LocalMod	KOKOSde	Self-hosted content moderation with prompt injection, toxicity, PII, and NSFW detection. 100% offline.
Guardrails AI	Guardrails AI	Validation rules & structured outputs using RAIL.
Lakera Guard	Lakera	Real-time prompt injection/jailbreak detection with near-real-time alerts.
Prompt Armor	Prompt Armor	Real-time detection and filtering of malicious prompts.
HiddenLayer AIM Security	HiddenLayer	AI application monitoring, real-time threat detection, and zero-trust access controls.
CalypsoAI Moderator	CalypsoAI	Commercial tool with audit trails, malicious code detection, and data loss protection.

---

Org	Name	Description
ISACA	AAISM™	AI Security Management (CISM/CISSP req.).
ISC2	Building AI Strategy Certificate	Strategy, governance, risk.
Practical DevSecOps	CAISP	Hands-on certification with labs.
Securiti	AI Security & Governance	Governance, privacy, compliance.

---

Name	Date	Location	Description
DEF CON	Aug 2025	Las Vegas	AI Village & GenAI red team challenges.
Black Hat USA	Aug 2025	Las Vegas	AI Security Summit & trainings.
RSA Conference	Apr-May 2025	San Francisco	AI security tracks, expo.
AI Risk Summit	Aug 19-20, 2025	Ritz-Carlton, Half Moon Bay, CA	Security executives, AI researchers, and policymakers discuss adversarial AI, deepfakes, and regulatory challenges.
GCSCC AI Cybersecurity Conference 2025	2025	Oxford, UK	Securing the Cyber Future: Cyber Resilience in the Age of AI and Geopolitical Uncertainty.
AI Security & Privacy Conference 2025	2025	TBD	400+ CISOs and C-Level Executives with expert-led discussions and case studies.
Cyber-AI 2025 Conference	Sep 1-4, 2025	Varna, Bulgaria	Four-day conference on cutting-edge advancements in cybersecurity and AI.
AI Village	Ongoing	Virtual/Various	Community, meetups, and CTFs.

---

Name	Topic	Description
Ignore This Title and HackAPrompt	Prompt Injection	EMNLP'23; taxonomy of prompt hacking.
SelfCheckGPT	Hallucination	Self-consistency for hallucination detection.
Survey on Model Extraction Attacks (2025)	Model Security	Survey of extraction attacks/defenses.
SECURE Benchmark	Cybersecurity	Multi-dataset security evaluation suite.
TruthfulQA	Safety	Truthfulness under misconceptions.
ToxiGen	Safety	Toxicity dataset & benchmarks.
In-The-Wild Jailbreak Prompts Dataset	Jailbreak	15,140 prompts with 1,405 jailbreak prompts from Reddit, Discord, websites (2022-2023).
JailbreakBench	Jailbreak	Open-source robustness benchmark with 200 distinct behaviors and jailbreak artifacts.
JailBreakV-28K	Jailbreak	28,000 jailbreak test cases for MLLMs (20K text-based, 8K image-based).
Forbidden Question Set	Safety	Curated dataset of forbidden questions across high-risk categories.
LLM Jailbreak + Safety Data	Jailbreak	~10K fine-tuning examples and ~3K adversarial prompts for chatbot safety.

---

Name	Author	Description
LangSmith	LangChain	Tracing + evals for LLM apps.
Weights & Biases	Weights & Biases	Experiment tracking & prompt management for LLMs.
Langfuse	Langfuse	Open-source tracing & cost monitoring.
Phoenix	Arize AI	Open-source eval/monitoring.
Helicone	Helicone	Proxy-based logging & analytics.
Dynatrace Davis AI	Dynatrace	AI-driven root cause analysis with multidimensional baselining and predictive analytics.

---

Name	Author	Description
PentestGPT	GreyDGL	GPT-powered pentesting assistant.
AI-penetration-testing	Mr-Infect	Curated offensive/defensive AI pentest techniques.
PentAGI	vxcontrol	Autonomous agent system for pentesting.
AI-OPS	antoninoLorenzo	Assistant for exploit dev & research.
HackSynth	aielte-research	Planner + summarizer pentest agent.
HexStrike AI MCP	0x4m4	150+ tools + AI agents automation.
Strix	usestrix	"AI hacker" agents; CLI & CI/CD.
BurpGPT	Burp Suite	Burp Suite extension integrating LLMs for enhanced vulnerability scanning and traffic analysis.

---

Model Context Protocol (MCP) security resources

Name	Author	Description
Lasso MCP Gateway	Lasso Security	First security-centric open-source solution for Model Context Protocol.

---

Name	Author	Description
awesome-llm-security	corca-ai	LLM Security resources.
awesome-gpt-security	cckuailong	Security tools & cases for GPT apps.
awesome-llm-cybersecurity-tools	Tenable	LLM tools for cybersecurity.
Awesome-LLMSecOps	wearetyomsmnv	LLM SecOps lifecycle & threats.
awesome-llm-supply-chain-security	ShenaoW	Supply chain security resources.
awesome-MLSecOps	RiccardoBiosas	MLSecOps tools & best practices.
awesome-hallucination-detection	EdinburghNLP	Hallucination detection papers.
oss-llm-security	kaplanlior	Curated list of open-source LLM security tools including EasyJailbreak, fast-llm-security, and more.

---

Name	Host/Author	Description
AI Security Podcast	Ashish Rajan & Caleb Sima	Vendor-neutral AI security conversations.
The AI Fix Podcast	Graham Cluley & Mark Stockley	Deepfakes, policy, and security.
Smashing Security	Graham Cluley & Carole Theriault	Weekly infosec pod with AI topics.
Resilient Cyber Newsletter	Chris Hughes	AI, supply chain, cloud, AppSec.

---

Cybersecurity and AI security YouTube channels

Name	Focus	Description
PowerDMARC	Email Security	Email authentication, DMARC, spoofing, phishing, and fraud tactics.
John Hammond	General Cybersecurity	1.28M subscribers; CTF challenges, hacking tutorials, and real-time problem-solving.
The Cyber Mentor	Ethical Hacking	Practical ethical hacking, penetration testing, and step-by-step tutorials.
NetworkChuck	Networking & Security	Exploring cybersecurity, networking, and technology concepts.
Hak5	Hacking Tools	Cybersecurity tools, privacy, tech gadgets, and entertaining tutorials.
MalwareTech	Malware Analysis	Deep-dive malware analysis, cybersecurity research, and threat intelligence.
David Bombal	Network Security	Ethical hacking, network security, and certifications.
LiveOverflow	Binary Exploitation	Low-level security, reverse engineering, and CTF writeups.
CyberRisk TV	AI Security	Black Hat 2025 coverage with focus on AI security, agentic AI, and trust.

---

Name	Author	Description
Rez0's AI Security Blog	Joseph Thacker	AI hacking fundamentals & techniques.
Simon Willison's Blog	Simon Willison	Prompt injection & agent security.
Lakera AI Blog	Lakera Team	GenAI security thought leadership.
Anthropic Transparency Hub	Anthropic	System cards & red team reports.
OpenAI Red Teaming Network	OpenAI	Red teaming docs & invites.
MLSecOps Community	Community	Best practices & community.
OWASP GenAI Security Project	OWASP	Global community-driven initiative for GenAI security guidance and resources.
OWASP AI Security Solutions Landscape	OWASP	Landmark guide outlining key risks and critical controls for securing LLMs and GenAI applications.
Lasso Security Blog	Lasso Security	Resources on LLM & AI cybersecurity, MCP security, and red teaming.

---

Name	Platform	Notability
Simon Willison	Twitter/X / Blog	Prompt injection & agent security.
Joseph Thacker (rez0)	Twitter/X / Blog	Prolific AI vuln research & guides.
Lakera Team	Twitter/X	Gandalf & Lakera Guard creators.
NVIDIA AI Red Team	Twitter/X	Team behind garak and practical security guidance.
Microsoft AI Red Team	Twitter/X	PyRIT & public red teaming lessons.
Steve Wilson	LinkedIn	OWASP Top 10 for LLM Applications Project Lead.
Ads Dawson	LinkedIn	Technical Lead & Vulnerability Entries Lead for OWASP Top 10 LLMs.

---

1. Purpose: Collect AI/LLM security & prompt-injection resources. Prefer open-source/community content.
2. Out of Scope: Ads, closed-source/proprietary, trials/freemium, or items needing private details.
3. Relevance: Must directly relate to AI/LLM security, jailbreaks, red teaming, model/app security.
4. No Duplicates: Avoid redundant entries.
5. Thought Leaders: Prefer figures tied to content/tools listed here.
6. Accuracy: Authors can open issues/PRs to update their entries.
7. Books: Paid books allowed for educational value.

How to contribute

• Fork → Branch → Edit README.md → Open PR with a clear description.
• See GitHub's Quickstart: Contributing to Projects.

---

To the extent possible under law, the contributors have waived all copyright and related or neighboring rights to this work.

---

⬆️ Back to top