Threat-model the tools, not just the prompts.

A team builds an assistant to classify customer requests and update a service platform. The prompt says to follow policy. Then the application gives it broad access to read customer records and issue updates. That tool permission is a more consequential design decision than the style of the system prompt.

The decision behind the technology

The model processes untrusted material: emails, PDFs, websites and retrieved documents. Prompt injection can persuade it to misuse the tools if the application grants excessive functionality, permissions or autonomy. System instructions are not a security boundary that substitutes for identity and authorization.

A better way to frame the work

Begin with asset and action inventory. Identify what each tool reads or changes, whose identity is used, and which operations are reversible. Restrict the allowed inputs and outputs, use deterministic policy gates, and require approval for high-consequence actions. Build logs that allow an investigator to reconstruct what happened.

Put the idea under realistic pressure

Inject malicious text into a sample supplier PDF requesting the agent to send data to an unrelated endpoint. Confirm the tool layer rejects any unsupported operation. Then try duplicate requests, role changes and failed writes. A successful defense cannot rely solely on the model saying no.

When the recommendation changes

Read-only agents and low-impact suggestions may need lighter controls. An agent that changes financial terms or accesses sensitive records needs a much stronger architecture. Autonomy is not a binary product feature.

What to decide before moving forward

Before shipping, document allowed tools, execution identity, secret boundaries, human approval events, idempotency, audit, incident response and negative test results. Review the controls whenever tools are added.

References and context

OWASP Top 10 for LLM Applications; NIST AI RMF Generative AI Profile.