Over the years, I’ve avidly followed automated penetration testing and its evolution, trying to determine whether it had the maturity to replace human pen testers outright. Progress was glacial and iterative, with many flaws and downsides, but also significant potential. In the past year, with a flurry of releases of AI-based pen testing tools, both open-source and commercial, I’ve had the opportunity to test many of them in depth, in side-by-side comparisons with human pen testers.