# Engineering autonomy is a control design problem. > Before an AI agent can act on an order, payment or customer record, someone needs to decide what it may do, how approval works and how to undo a mistake. Canonical URL: https://marmation.com/review/engineering-autonomy-is-a-control-design/ By Vikram Roy · 2026-11-12 The business value of an agent depends on what it can finish safely. A system that proposes a useful credit and a system allowed to issue one carry different responsibilities. Decide where that authority changes hands, and what happens when the result is uncertain. An employee asks an assistant to resolve an invoice mismatch. The assistant gathers purchase order context, identifies a likely discrepancy and suggests a credit. On a prototype screen, a friendly 'Apply change' button makes the experience feel finished. The difficult part is invisible: whose credentials authorize the action, what exactly will be written, and what happens if the API times out after it succeeds? Autonomy is not simply a model choosing among tools. It is an application allowing a component whose answers can vary to request actions inside a business system that must apply fixed rules. ## Start with authority, not tool selection For every proposed action, name the identity under which it will execute, the resources it may read, the records it may change, and the conditions under which it may do so. The model should not gain broader permissions because a user puts persuasive instructions in a document or a chat message. A read-only retrieval operation may be safe for wider delegation if source permissions are respected. Creating a draft ticket is different from sending a customer email. Updating a quote is different from issuing a refund. The design needs those distinctions before deciding whether the agent will have a single tool, a planning loop or a multi-agent orchestration system. ## Give the workflow explicit states Give each stage of the task a name and rules for what can happen next. Engineers call this a state machine. It might contain: received, evidence gathered, proposed, review required, approved, save pending, saved, final result awaiting confirmation, failed and closed. Each transition has an owner and a rule. The UI should show the state as clearly as the model's reasoning. A reviewer must know which action is merely proposed, what will change if approved, what evidence supports the change and whether the approval has already been used. Final policy enforcement belongs in the application, not the natural-language prompt. ## Network failures are an autonomy problem Suppose the destination API successfully creates a credit, but the response is lost. An agent that sees a timeout may repeat the request and create a second credit. This is not a hallucination. It is a distributed-systems failure. Use idempotency keys where supported to identify a repeated request, keep a record of the transaction’s stage, and link related requests with correlation IDs. Where the destination cannot guarantee a single effect, design a check of what actually happened (reconciliation) or an authorized way to reverse the effect (compensation). Retry rules should be documented per action. Sensitive updates may need a human review checkpoint before the first commit and an explicit operator escalation before retrying uncertain outcomes. The system also needs rate limits, maximum spend or action bounds, exception queues and alerts for repeated failures. A model's confidence in its own plan cannot override these controls. ## Test the malicious instruction, not just the happy path Place a simulated prompt injection in an incoming email instructing the agent to send payment data to a different account. The correct system behavior is not merely to answer that it should ignore the instruction. It is to prevent any unauthorized destination from being passed to the write API. Test a user whose permission was revoked midway through a workflow. Test two pending approvals on the same item. Test a failed external API and a model provider outage. Test a human who corrects the proposed value. Test access across organizations or tenants when the system is multi-tenant. ## A useful autonomy ladder The same business workflow can support different degrees of delegation. Begin with read-and-suggest. Then drafts that the user reviews. Then bounded actions with strong deterministic validation, reversible changes and recorded approvals. Allow higher autonomy only if repeated evaluations, operational monitoring and consequences justify it. There is no universal safe level for every organization. A low-stakes draft meeting summary and an irreversible financial action have different risk profiles. **The decision:** Let the agent propose actions within agreed limits. Require the application to enforce who may approve each change and whether it can proceed. ### References and limits [OWASP Top 10 for LLM Applications](); [NIST Generative AI Profile](). Technical recommendations vary by system risk; not a claim of certified safety. ## Sources - [Generative AI Profile for AI RMF]() — NIST. Reference governance and risk design principles for gen AI. Risk management reference for generative AI Limit: This is a risk-management reference, not a certification or evidence that Marmation has been audited or complies with it. - [Top 10 for LLM Applications]() — OWASP. LLM risks such as prompt injection, excessive agency and data disclosures. Threat categories for LLM systems Limit: This is an engineering security reference, not evidence of an independent audit of Marmation or another system. ## Related reading - [Ideas that stand up to implementation.](https://marmation.com/review/) - [From working prototype to dependable system](https://marmation.com/review/engineering/) - [Design the action boundary before designing the agent.](https://marmation.com/review/designing-an-agent-with-bounded-actions/)