SecurityScorecard recently announced the launch of the world’s first AI TPRM Benchmark. The cybersecurity leader designed this standardized framework to evaluate how artificial intelligence models manage complex third-party risk tasks. Furthermore, the testing system measures how effectively AI models withstand adversarial manipulation and supply chain attacks.
Indeed, the platform leverages one of the world’s largest third-party risk datasets to measure operational performance. Consequently, organizations gain an objective method to assess model capabilities before deploying AI across critical workflows.
Today, enterprises rapidly introduce automated AI models into daily security and risk management workflows. However, security leaders must determine whether these models remain trustworthy under active adversarial conditions. Most traditional testing models do not account for real-world conditions of third-party risk management. Here, AI needs to be able to analyze intricate telemetry and assess possibly misleading vendor information.
Therefore, the TPRM Benchmark addresses this critical gap by using real-world supply chain scenarios. The framework combines two vital evaluation dimensions. First, it measures task execution capabilities like exposure triage, breach correlation, and risk analysis. Second, it tests model resilience against prompt injection, jailbreaks, and manipulated inputs.
“Putting an AI agent into your vendor-risk program without testing it is like giving a new employee privileged access on day one and hoping for the best,” said Dr. Aleksandr Yampolskiy, CEO and Co-Founder of SecurityScorecard. “The TPRM Benchmark answers two questions every security leader should be asking: can this model do the work, and can it be turned against me? If you can’t answer both, you don’t have an AI advantage. You have a new attack surface.”
Evaluating AI Task Performance and Operational Resilience
The framework evaluates AI models through a rigorous two-part methodology. In the first pillar, the TPRM Benchmark measures model performance across exposure triage, vendor concentration, and remediation planning. Each task utilizes actual SecurityScorecard telemetry to reflect genuine operational environments. Additionally, the system rotates task inputs and protects answer keys to prevent benchmark contamination.
The evaluation assigns an autonomy readiness level from L0 to L4 based on repeated performance. Consequently, this grading helps security leaders determine where human oversight remains necessary.
In the second pillar, the framework tests how models handle deceptive vendor information and direct cyber attacks. Adversaries often manipulate self-attestations, questionnaire responses, and inbound communications to bypass security controls. Therefore, the testing suite subjects models to prompt injection attacks and grades resistance on an A-to-F scale.
“Our initial benchmark runs show open-weight models closing the gap with leading frontier models on supply chain security tasks, but several of those same models remain highly susceptible to adversarial attack,” said Dr. Mike Mylrea, CISO and Chief AI Officer at SecurityScorecard. “As AI increasingly takes on critical security functions, organizations need to know not just which models perform best at each task, but which ones hold up when someone is actively trying to manipulate them.”
“There are many AI models on the market that deliver the results they promise, but very few adhere to the cybersecurity standards needed for real industry adoption, and that gap is exactly why we need an evaluation platform like this one. I think of a good benchmark like a race track: it shows you which models have real problems under the hood, and it reveals the true limits of what’s possible, which is what makes this work academically meaningful, about scientific discovery, not just about picking the best commercial model off the shelf,” said Dr. Yesha, the Director for the Machine Learning and AI program at IDSC. “It also matters enormously for organizations trying to use AI in third-party risk workflows specifically. It’s not enough to know a model can complete the task. Done right, a benchmark like this is how you get the confidence to actually deploy AI in these workflows, knowing it does the job well and that its security can be trusted.”
“TPRM Bench stands out by testing AI against real supply chain scenarios grounded in empirical security data,” said Yevgeniy Dodis, Professor of Computer Science at New York University and Fellow of the IACR (International Association for Cryptologic Research). “Its rigorous, reproducible framework helps bridge the gap between benchmark performance and real-world security as AI adoption in critical infrastructure accelerates.”
“You can’t secure an attack surface you haven’t measured, and TPRM Bench gives security teams a shared, transparent way to evaluate models on capability and attack resistance together,” said Manoj Nair, Chief Technology & Innovation Officer at Snyk. “In our own Snyk research across 3,000 enterprise environments, 74.5% of those with a detected model, reference at least one that scores ‘high’ on an attacker goal under adversarial testing.”
“Preempting the next cyber attack means proving your defenses before they’re put to the test,” said Peter Lee, CEO of SimSpace. “Trust in AI must be earned through continuous testing in realistic, production-like conditions. SecurityScorecard’s TPRM Benchmark helps build that trust by measuring whether AI can perform critical tasks and resist manipulation. For security leaders, validating AI is essential for deciding how much autonomy to give agents and for pinpointing when humans need to remain in control.”
“AI is now part of both attacker and defender tooling, so unexpected model behavior can quickly become an incident for one side or an opportunity for the other. At GreyNoise, we’ve seen attackers’ AI use leave identifiable tells, and we’ve seen defenders’ models make mistakes analyzing packets and TTPs,” said Corey Bodzin, Chief Product Officer of GreyNoise. “Objective, transparent measurement like SecurityScorecard’s TPRM AI Benchmark puts evidence of model performance on the table, so defenders can adopt AI with a clear view of its strengths and limits.”
To explore how Security Operations Centers (SOC) play a crucial role in defending against modern cyber threats, read our latest SOC News.
