Building an AI Agent for HalCTF at DEF CON 34
This past weekend, I attended DEF CON 34, the hacking conference. While I was there, I participated in the HalCTF event, where you build an AI agent and the agent plays the Capture The Flag (CTF) game.
This whole journey started last Thursday around 4:20 a.m., when I got to the airport way too early for a flight that was supposed to take off at 6:20 a.m. While I was waiting to board, I went online to look up information about DEF CON: which talks would be interesting, and which villages would have events or talks worth attending. I saw that the AI Village would have this HalCTF, where participants were asked to make an AI agent and the agent would be the actual player, solving the challenges.
I thought this would be a very good opportunity to experiment with agent-design concepts I had been reading about and was curious to try out. I also thought it would be a great opportunity to meet other people engineering harnesses to squeeze performance out of non-frontier models and producing agents capable of complex tasks like hacking and playing a CTF. So that Thursday morning, I immediately started working on my agent design.
Unfortunately, I made some design decisions based on incomplete information, or on my misinterpretation of information published by the game organizers which made my agent a bit more complicated and harder to debug. For example, I assumed that we would upload an agent into the CTF infrastructure, that it would be able to see multiple challenges at the same time, and that it would have to decide which challenge to tackle first, second, third, etc., in order to maximize its chances of solving challenges and scoring the highest number of points. But that assumption turned out to be incorrect when I actually started the game: the agents would be playing one challenge at a time, assigned manually by the human participant from outside the game environment.
My first design decision was to build a multi-agent system consisting of an orchestrator agent, which I named the captain, and sub-agents that would be specialists in different classes of CTF problems, which I named the solvers. The rationale was that this split would help me make better use of a limited context window by loading only the stuff needed for a specific job at a time: the captain would be my expert in CTF strategy and game mechanics, while the solvers would be experts in domains like web, crypto, reversing, and so on.
+--------------------------------------------------------------------+
| TEQUILA: CTF agent |
| |
| +------------------+ |
| | [captain] | |
| | strategy, routing| |
| | deadlines, state | |
| +---------+--------+ |
| | |
| +----------------+-------------+-----------+-------------+ |
| | | | | |
| v v v v |
| +-------------+ +-------------+ +-------------+ +---------+ |
| | web [solver]| | crypto | | reversing / | | other | |
| | | | [solver] | | pwn [solver]| | solvers | |
| +-------------+ +-------------+ +-------------+ +---------+ |
| |
+--------------------------------------------------------------------+
|
v
Target
To develop my agent’s expertise, I thought it would be a good idea to sample and study CTF write-ups found on the internet, and try to distill the general workflows that successful challenge solves tend to follow in the different categories, as well as general knowledge of technologies, vulnerabilities, attacks, and tools based on the types of puzzles and problems often seen in Capture the Flag. I started by creating a library of 400 CTF write-ups, which later grew to 1,000 write-ups. This was my reference library for how to successfully solve CTF challenges.
A funny detail: I used an AI agent to help me process the write-ups and some of the web pages had hidden prompt injections. Luckily, my agent noticed the prompt injections, ignored the instructions, and simply highlighted them for me as a warning. If you are thinking about sampling write-ups from the internet, be careful in how you process the contents so that your agent does not fall for these prompt injections and uploads your .env to some hacker on the Internet.
I also used progressive disclosure as a context-engineering technique. Rather than load every piece of knowledge into every prompt, I created small knowledge cards that could be loaded when they were relevant to the kind of challenge the agent was working on.
TEQUILA runtime instructions
├── coordinator.py:PLANNER_PROMPT
│ └── captain instructions:
│ strategy, routing, checkpoints, stages, deadline, submission
│
├── coordinator.py:WORKER_PROMPT
│ └── shared solver contract:
│ scope, evidence rules, tool protocol, submit, handoff
│
├── profiles.py:specialist_profile(role)
│ └── prompts/specialists/
│ ├── web.md: web triage and exploit workflow
│ ├── crypto.md: crypto analysis and solver workflow
│ ├── reversing.md: binary and verifier workflow
│ ├── pwn.md: binary-exploitation workflow
│ ├── ai.md: AI challenge workflow
│ ├── network.md: network and protocol workflow
│ └── ...: other specialist profiles
│
├── profiles.py:specialist_knowledge(role, challenge_text)
│ └── prompts/knowledge/ai/
│ ├── prompt_indirect_boundary.md
│ ├── agent_tool_boundary.md
│ ├── rag_retrieval.md
│ ├── memory_multiagent_boundary.md
│ ├── model_classifier_evasion.md
│ └── infrastructure.md
│ └── loads up to two cards for AI challenges
│
└── profiles.py:challenge_knowledge(title)
└── prompts/knowledge/challenges/
├── kanto_cerulean_cave.md
├── pantheon_hydra_signature.md
├── odyssey_cattle_helios.md
└── ...: other challenges knowledge
The next important components of my agent design were its memory system and tools. For memory, I debated between greppable files and a lightweight SQLite database. I chose SQLite because it gave the agent durable working memory without turning every note into unstructured text.
Confirmed observations went into facts, while possible solution paths stayed as hypotheses until tested. I also separated model claims from confirmed facts: a solver could suggest an explanation, but the captain would only rely on it once it was tied to an observation. The database could retain more evidence than would fit in a model’s working context, then reconstruct a smaller relevant view when needed.
Tequila SQLite memory
runs
└── one assigned challenge / one agent run
│
├── observations
│ └── immutable tool output and saved evidence
│ │
│ v
│ claims
│ └── model interpretation tied to an observation
│ │
│ v
│ facts
│ └── captain-approved claim:
│ safe to use for later planning
│
├── hypotheses
│ └── possible solution paths
│ proposed → tested → confirmed / disproved
│
├── agent_sessions
│ └── each solver lease:
│ role, objective, action budget, status
│
└── stage_plans
└── multi-stage solve state:
current stage, completion criterion, next dependency
The first tool surface was deliberately tiny: shell access. That was enough to prove the basic loop, but it was not the final design.
Over the next 20-something hours, I created the first version of the agent. To test it, I signed up for a couple of free CTFs online that were no longer live competitions but whose games were still available for educational purposes.
I noticed that my agent was much better at solving web challenges than crypto, binary reversing, or pwnage challenges. I measured these practice runs in solver actions: one command, file operation, submission, or handoff at a time. The practice results reflected that difference. One web challenge was solved in 12 actions after the supplied source revealed an unauthenticated deserialization path. A pwn challenge succeeded, but took 36 actions. A crypto padding oracle was correctly identified in seven actions, but could not finish within a 120-second tool lease. I later encoded this into the captain’s strategy, so it would play to the agent’s strengths and go for the categories in which it was strongest first, trying to score points early and get first-blood points. By Friday morning, in my small practice set, I had a working agent that was able to solve roughly 70% of the challenges without intervention, which I thought was pretty good.
The start of the game was pretty rough. As some readers will know, the DEF CON network is one of the most hostile networks in the world. On Friday, it was experiencing all sorts of difficulties, which impacted the game because the we could not connect correctly to cloud-based infrastructure and inference services. I decided to wander off and explore DEF CON while the network issues were resolved, and I was surprised to see how much the event has grown in the last few years.
Later in the day, the network issues were resolved. When I realized the game had already started, it had been going on for about 45 minutes and some teams had already scored points. The first teams at the top of the leaderboard had about 450 points. I rushed back to the AI Village area, uploaded the first version of my agent, and started it in the first CTF environment: the one designed for smoke-testing agents, ensuring containers were correctly formed, and confirming that the agent could connect to the inference service and the MCP server provided by the organizers for interacting with the challenge board.
Here I found my first lesson. I had incorrectly assumed all agent output would be displayed publicly to the rest of the participants. So I deliberately created my agent not to output log information, because I did not want to leak details about how it worked during the game and enable other participants to steal information or copy some of my strategies. This assumption was incorrect: the game organizers’ platform made standard output and logs private, so only participants could see their respective logs.
This became evident when I loaded the first version of the agent into the smoke-test environment. It was not working, and I could not really figure out why. Because it was not printing any output, I had no way to debug it. I had to quickly iterate and create the next version of my agent that would print useful logs so I could debug why it was not running correctly. Once I had useful logs, I was able to figure out some of the issues with the agent and make the next iteration. It probably took me five iterations to finally have an agent that could correctly run and interact with the game environment. Then I moved on to the first challenge that awarded points.
One of the big challenges during the game was that only so many agents could play at the same time because compute resources were limited. You could upload your agent and it would enter an execution queue, sitting there until resources were freed up and a turn was available. Then your agent could start its run.
The environment offered three model options: Llama, Gemma, and Qwen. The tradeoff was not simply model quality. Model choice affected scoring multiplier, queue behavior, concurrency, and expected reasoning capability. I chose Gemma because I thought its reasoning capability and higher concurrency would let me make steady progress rather than wait for a larger model.
Your agent could run for up to one hour, and at the one-hour mark the game infrastructure would stop the run whether it got the flag or not. As the weekend progressed, more and more people got interested in the game, so the system became congested and the run queue got very long: perhaps 45 minutes to an hour before an agent would actually run. That meant people had time to get creative and explore more of the game mechanics.
One thing a few participants found was that the game was structured as five or six CTFs, each with its own set of challenges. Players had to upload their agent into each CTF in order to run it. As I was improving my agent and iterating, I noticed that some times I could enqueue two runs simultaneously as long as they were in different CTFs but I wasn’t sure what exactly was the rule. Later, I figured out that this was being allowed because the hashes of the agent container images were different. As part of the game strategy, I would produce different containers of the same agent, manipulated to have different hashes, upload them into the different CTFs, and queue them so that I always had a run queued in each CTF. I called this rule Always Be Competing, or ABC.
That is a bit about the game and its mechanics. Now, what are some of the things that did not work so well and that I spent a lot of time fixing for my agent?
One of the biggest lessons was that an agent can appear weak at security reasoning when the real failure is architectural. It might lose a fact, fail to express a tool call, be unable to construct a script, misread an API error, or keep working after it has already won. Fixing those problems often mattered more than adding another piece of security domain knowledge.
I would say a lot of it had to do with tool use. The agent struggled with correctly using the shell tool, constantly making quotation mistakes and spending a lot of time trying to fix them. Another issue was that the captain would monitor the solvers and allow them only a limited number of actions to solve a challenge. If a solver did not solve the challenge within that action budget, the captain would stop it. I saw that the captain was prematurely stopping solvers even when they were making good progress.
One early fix was to increase the action budget. I came to think of that budget as a lease: a solver needed enough uninterrupted time to build an artifact, run related experiments, and interpret the results, while the captain needed to regain control at meaningful checkpoints instead of after every action. One capable solver was often more useful than several shallow solvers working in parallel, because parallel work duplicated setup, consumed model budget, and complicated the investigation.
Later, organizers shortened the maximum runtime from one hour to 15 minutes. At that point, the external deadline became the real budget. I removed the cumulative action cap and had the captain interrupt only when a solver was clearly stuck, then start a new solver on a different hypothesis.
I also reserved the final minute of each run for checkpoint recovery, submission, and graceful finalization. There was no point beginning new exploration when there was too little time left to finish it.
To address the tool issues, I ended up with a very simple set of tools:
shell: run normal Linux commands and solver programs. The agent needed a real shell, not a narrow command allowlist that rejected useful syntax.write_file: create and extend scripts without forcing multiline code through shell quoting, heredocs, and JSON escaping.read_file: inspect the exact script or saved evidence after writing it, so the agent could verify, compile, and debug what it actually produced.
Keeping the tool surface this small helped the agent avoid getting confused about which tool to use for a task. The agent could often reason about what a script needed to do, then fail at the mechanical step of expressing it safely, so the tools evolved to be more ergonomic or intuitive for the agent.
I also found that my agent had a few capability gaps in the crypto category. For some crypto challenges, it was necessary to gather a bunch of information from the target in order to do analysis, solve the math, and obtain a key or an answer. When the agent spent most of its time discovering the challenge and creating a tool, it would not have enough time left to run it enough times to gather the necessary information. For cases like that, I started including general-purpose helpers in the image: small data collectors, local constraint-solving tools, and compact HTTP clients. The goal was not to ship a solver for a specific challenge. It was to stop wasting scarce runtime rebuilding ordinary infrastructure before the agent could begin the actual analysis.
I also rewrote the prompts and instructed the agent to communicate in a deliberately plain, “caveman” style. Dense role-playing prose was not helping the model act. Direct instructions about the next experiment, the expected evidence, and the stopping condition worked better.
Submission rate limits also made me change the agent. A valid solve could be wasted by careless retries, so I added stricter validation for candidates before submission and had the agent back off for 60 seconds, then 120 seconds, after an HTTP 429 rate-limit response.
By the later releases, ultimately v0.2.27, the agent was no longer struggling with the earlier tool and harness issues, and it was able to focus on actually solving the challenges. It started to get pretty good results, consistently solving some of the harder challenges, the ones that awarded between 400 and 500 points. But unfortunately, this happened too late in the event, and by the time my agent was good enough to solve high point challenges, the game was already ending. Still, the agent ended up in the top 20 out of more than 200 teams.
The main lesson was that the agent improved less by acquiring new domain specific knowledge than by becoming operationally reliable. Better evidence handling, durable handoffs, script construction, submission behavior, and graceful shutdowns turned more of its reasoning into actual solved challenges.
Thanks and shout-outs to the organizers, especially Cyduck and Zach, and to the other players who hung out at the end and shared ideas and lessons learned: AbliteratedEdgeModel, devins-revenge, feetpics.ai, beymax and aisafe.