Immersive Labs Unveils New ‘Dark Side of GenAI’ Report about How People Trick Chatbots into Exposing Company Secrets
2024年5月21日 - 7:00PM
ビジネスワイヤ(英語)
Analysis of prompt injection techniques reveals
organizations are at risk as GenAI bots are susceptible to attacks
by users of all skill levels, not just experts
Immersive Labs, the global leader in people-centric cyber
resilience, today published its “Dark Side of GenAI” report about a
Generative Artificial Intelligence (GenAI)-related security risk
known as a prompt injection attack, in which individuals input
specific instructions to trick chatbots into revealing sensitive
information, potentially exposing organizations to data leaks.
Based on analysis of Immersive Labs’ prompt injection challenge*,
GenAI bots are especially susceptible to manipulation by people of
all skill levels, not just cyber experts.
Among the most alarming findings was the discovery that 88%
of prompt injection challenge participants successfully tricked the
GenAI bot into giving away sensitive information in at least one
level of an increasingly difficult challenge. Nearly a fifth of
participants (17%) successfully tricked the bot across all levels,
underscoring the risk to organizations using GenAI bots.
This report asserts that public and private-sector cooperation
and corporate policies are required to mitigate security risks
posed by the extensive adoption of GenAI bots. Leaders need to be
aware of prompt injection risks and take decisive action, including
establishing comprehensive policies for GenAI use within their
organizations.
“Based on our analysis of the ways people manipulate GenAI, and
the relatively low barrier to entry to exploitation, we believe
it’s imperative that organizations implement security controls
within Large Language Models and take a ‘defense in depth’ approach
to GenAI,” said Kev Breen, Senior Director of Threat Intelligence
at Immersive Labs and a co-author of the report. “This includes
implementing security measures, such as data loss prevention
checks, strict input validation and context-aware filtering to
prevent and recognize attempts to manipulate GenAI output.”
Key Findings from Immersive Labs “Dark Side of GenAI”
Study
The team observed the following key takeaways based on their
data analysis, including:
- GenAI is no match for human ingenuity (yet): Users
successfully leverage creative techniques to deceive GenAI bots,
such as tricking them into embedding secrets in poems or stories or
altering their initial instructions, to gain unauthorized access to
sensitive information.
- You don’t need to be an expert to exploit GenAI: The
report’s findings show that even non-cybersecurity professionals
and those unfamiliar with prompt injection attacks can leverage
their creativity to trick bots, indicating that the barrier to
exploiting GenAI in the wild using prompt injection attacks may be
easier than one would hope.
- As long as bots can be outsmarted by people, organizations
are at risk: No protocols exist today to fully prevent prompt
injection attacks. Cyber leaders and GenAI developers need to
urgently prepare for – and respond to – this emerging threat to
mitigate potential harm to people, organizations, and society.
“Our research demonstrates the critical importance of adopting a
‘secure-by-design’ approach throughout the entire GenAI system
development life cycle,” added Breen. “The potential reputational
harm to organizations is clear, based on examples like the ones in
our report. Organizations should consider the trade-off between
security and user experience, and the type of conversational model
used as part of their risk assessment of using GenAI in their
products and services.”
The research team at Immersive Labs consisting of Dr. John
Blythe, Director of Cyber Psychology; Kev Breen, Senior Director of
Cyber Threat Intelligence; and Joel Iqbal, Data Analyst, analyzed
the results of Immersive Labs’ prompt injection GenAI Challenge
that ran from June to September 2023. The challenge required
individuals to trick a GenAI bot into revealing a secret password
with increasing difficulty at each of 10 levels. The initial sample
consisted of 316,637 submissions, with 34,555 participants in total
completing the entire challenge. The team examined the various
prompting techniques employed, user interactions, prompt sentiment,
and outcomes to inform its study.
For more about these and other insights, access the report today
at: https://www.immersivelabs.com/dark-side-of-genai-report/.
*Immersive Labs Prompt Injection Challenge, June-September
2023
About Immersive Labs
Immersive Labs is the leader in people-centric cyber resilience.
We help organizations continuously assess, build, and prove their
cyber workforce resilience for teams across the entire
organization, from front-line cybersecurity and development teams
to Board-level executives. We provide realistic simulations and
hands-on cybersecurity labs to evaluate individual and team
capabilities and decision-making against the latest threats.
Organizations can now prove their cyber resilience by measuring
their readiness compared to industry benchmarks, building team
capabilities, and demonstrating risk reduction and compliance with
data-backed evidence. Immersive Labs is trusted by the world’s
largest organizations and governments, including Citi, Pfizer,
HSBC, the UK Ministry of Defence, and the UK National Health
Service. We are backed by Goldman Sachs Asset Management, Summit
Partners, Insight Partners, Citi Ventures, Ten Eleven Ventures, and
Menlo Ventures.
View source
version on businesswire.com: https://www.businesswire.com/news/home/20240521256421/en/
Jen Gaines immersivelabs@inkhouse.com