Anthony Spadaro is the founder and CEO of Smart Founder Lab, an agency building agentic AI software for nuclear power and energy operations.
getty
Every few weeks, we hear another confident prediction that autonomous agents will take over enterprises. But if you take that assertion and apply it to the industries I've spent the bulk of my career with—nuclear and energy production—you'll quickly find there's a brick wall.
The hardest part of agentic software in a regulated space has never been the intelligence—it's the system around the intelligence to keep it in its lane. That matters because teams usually think about agentic software like they do any other AI application—in terms of what it can do. But in regulated industries, we need to know what the agent should never be allowed to do. That's where I start thinking about the system design.
Constrain your allowable actions before you expand your intelligence.
With most agentic AI models, you want to give them room to run and see what they can do. In a regulated space, you want to do the opposite. These systems tend to operate within a tight, allowable set of actions, where the system is restricted by deterministic, hard logic, outside of the AI's decision making. An agent can think about what it would suggest, but the rules governing what's allowed live outside of the model, where they're auditable by a reviewer or regulator. It's slower to build, but that's what gets you from a demo to what an operator is willing to touch.
Make the system honest about what it doesn’t know.
An agentic AI system should be designed so that when it doesn't know something, it says so. I've found that embedding honest responses into the product, instead of masking uncertainty behind a confident but potentially wrong output, will turn the most skeptical operators into willing adopters.
Build for the auditor, not just the user.
For most consumer software, the end user is the only person that matters. In the nuclear and energy sectors, it's whoever will have to reconstruct, months from now, why a particular decision was made. Thus, every action an agent makes must be traceable to a reason, and that reason must be documented as the event itself takes place, rather than retroactively pieced together.
You can't tack auditability onto the end like a compliance tax; when done right, it's one of the single greatest selling points for what you provide. It's the thing that allows a cautious enterprise to hit "approve." The framework 10 CFR 50 Appendix B, which governs quality assurance requirements, is one of many. So is the human guidance on which the industry currently leans for control room interaction. These aren't hurdles for designers to circumvent. Rather, they're useful guidelines for what good software should be in the first place.
Earn autonomy one rung at a time.
The "agentic" bit suggests autonomy, and a highly regulated customer isn't yet prepared for that on day one. So, start out with advisory capabilities: The system makes a suggestion, and a competent human gives the final word. The system only earns more freedom as the model gains trust by building a history of getting it right and also being truthful about when it was wrong.
Organizations that try to jump right into full autonomy very often get shut out, not by the models but by the people on the hook when things go wrong. That isn't the tale that's currently being told about agents in the rest of the market, where the narrative is focused on reach and speed. In regulated industries, the best systems are understandable, bounded and unashamedly honest about their own boundaries.
If you're building an agentic AI system for any industry where mistakes aren't an option, start from what the agent can never do and proceed from there. This is the type of intelligent software these industries will actually let into their companies.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?