Prompt Injection Capture-the-flag – Red Team x AI

Joe Cooney • April 2, 2024

Red-team challenges have been a fun activity for PZ team members in the past, so we recently conducted a small challenge at our fortnightly brown-bag session, focusing on the burgeoning topic of prompt injection. 


Injection vulnerabilities all follow the same basic pattern – un-trusted input is inadvertently treated as executable code, causing the security of the system to be compromised.  SQL injection (SQLi) and cross-site scripting (XSS) are probably two of the best-known variants, but other technologies are also susceptible. Does anyone remember XPath injection? 


As generative models get incorporated into more products, user input can be used to subvert the model. This can lead to the model revealing its system prompt or other trade secrets, reveal information about the model itself which may be commercially valuable, subvert or waste computation resources, perform unintended actions if the model is hooked up to APIs, or cause reputational damage to the company if the model can be coerced into doing amusing or inappropriate things. 


As an example, entrepreneur and technologist Chris Bakke was recently able to trick a Chevy dealership’s ChatGPT-powered bot into agreeing to sell him a Chevy Tahoe for $1. Although the U.S. supreme court has yet to rule on the legal validity of a “no takesies backsies” contract (as an employee of X Chris is probably legally obligated to drive a Tesla anyway) it is not hard to imagine a future scenario with steeper financial consequences. 


For this challenge PZers were taking on Gandalf https://gandalf.lakera.ai/  – a CTF created by AI security start-up Lakera https://www.lakera.ai/ (Gandalf is doubtless a way for them to capture valuable training data for their security product). Gandalf progresses in difficulty from young and naive level 1 Gandalf, who is practically begging to give you the password, to level 8 – Gandalf the White 2.0, who is substantially more difficult to trick. 

We time-boxed the challenge to only 20 minutes, and a couple of people were able to beat Gandalf the White 2.0 in this time. Several PZers also found the challenge so absorbing they were still going an hour or more later. Some people found prompts that worked well for several levels, allowing them to rapidly progress to the higher levels of the challenge, only to hit a wall when their chosen technique stopped working. Others were beguiled into solving riddles that Gandalf seemed to be posing to them in the hope that it would give them clues to the secret word for each level. 


Overall, it was a fun and approachable challenge for anyone looking to become more familiar with the issue of prompt injection. 

Share This Post

Get In Touch

Recent Posts

Security, But Fun - Running A CTF Event for Your Dev Team

By Joe Cooney • April 3, 2025
Making cybersecurity fun and engaging with capture-the-flag (CTF) events—boost team collaboration, enhance security skills, and turn dry security practices into an exciting challenge!

Demelza Green, Featured on ABC's 'This Working Life' Podcast

January 16, 2025
We are excited to share that our Co-CEO, Demelza Green , was recently a guest on 'This Working Life' , a podcast by the Australian Broadcasting Corporation (ABC) hosted by Lisa Leong . During the episode, Demelza discussed the evolving landscape of hybrid work and how virtual reality (VR) is shaping the future of workplace collaboration. "Recording the podcast was a unique experience," Demelza shared. "I was sitting on a park bench next to the river in Mooloolaba. Despite my mum insisting I've never sounded more Australian, I wonder if listeners can spot my strong Kiwi accent, as I thought it was as strong as ever. It's funny how recording outside can change the sound of your voice." Demelza also responded to Lisa's request for pictures of teams working in VR: "Our team got dressed up and coordinated a round of thumbs-up just for Lisa!"  Listen to the full episode here: Managing Hybrid Work - This Working Life

Demelza Green Wins ARN Innovation Management Excellence Award

November 26, 2024
We are thrilled to announce that Demelza Green , our co-CEO, has been awarded the prestigious ARN Innovation Management Excellence Award at the 2024 ARN Innovation Awards. The ARN Innovation Awards celebrate outstanding achievements in the Australian IT industry, recognising individuals and organisations that drive innovation and contribute significantly to the technology sector. This accolade highlights Demelza's dedication to driving innovation within Patient Zero. "I am incredibly honoured to receive this award," said Demelza. "Innovation is a team effort, and this recognition reflects the hard work and creativity of the entire Patient Zero team."  Congr atulations to Demelza on this well-deserved award!

Hanieh Madad Wins Women in Digital Technical Leader of the Year Award

October 25, 2024
We’re pleased to share that Hanieh Madad, Senior Software Developer and Team Leader at Patient Zero, has been awarded the Women in Digital Technical Leader of the Year. This award recognises Hanieh’s dedication to her craft and her thoughtful approach to leadership within the tech industry. The judges highlighted Hanieh’s exceptional handling of a complex project, noting her skill in managing stakeholders, mentoring junior engineers, and her commitment to community contributions. In her acceptance speech, Hanieh shared, “I wouldn’t be standing here without my amazing team that I have had the privilege of working with. This award is as much theirs as it is mine.” At Patient Zero, Hanieh leads with a balance of technical expertise and thoughtful mentorship. Known for guiding complex projects to success, she consistently supports her team’s growth and development, making this recognition truly fitting. Congratulations, Hanieh, on this achievement and for the positive impact you continue to make.
More Posts
Share by: