Reinforcement learning for an autonomous pentesting agent
Can an open model, trained with reinforcement learning for one agent system, replace the frontier models that you rent through an API?
- Client
- Cybersecurity company
- Role
- Senior AI engineer
- Model
- Qwen 32B
- Methods
- Reinforcement learning, multi-agent systems (LangGraph)
The problem
The client has a system of several AI agents that does penetration tests with no person in the loop. It ran on frontier models through an API.
The question was simple. Could an open model, trained for this system only, do the same job?
What I built
- A pipeline that turns the long traces of the agents into training data.
- A fine-tune of Qwen 32B with reinforcement learning on that data.
- An evaluation against two references: the same model before training, and the frontier models.
Results
The training worked. The fine-tuned model was clearly better than its base on the client's tasks.
But in the same months, the frontier models improved faster than the fine-tune could follow. The client chose a different direction.
What I learned
- Compare with the best frontier model at every milestone, not only with the base model. The gap that counts is the gap to the frontier.
- Agree on the stop criteria at the start, as numbers. Then the decision to continue does not depend on momentum.
- Prove the gain on a small pilot before you pay for more compute.
- Know when fine-tuning pays. It pays when privacy, deployment on your own hardware, latency or cost at scale are hard requirements. When the requirement is pure capability, the frontier models usually win.