# Threat-model the tools, not just the prompts.

> An AI agent with business-system access inherits the risk of the actions it can take.

Canonical URL: https://marmation.com/review/threat-modeling-tool-using-ai-agents/

By Vikram Roy · 2026-11-12

An assistant that can change business records creates risks beyond a misleading answer. Ask what damage each connected tool could cause, whose permission it uses and what would stop a malicious document from triggering that action. That exercise is threat modeling.

A team builds an assistant to classify customer requests and update a service platform. The prompt says to follow policy. Then the application gives it broad access to read customer records and issue updates. That tool permission is a more consequential design decision than the style of the system prompt.

## Treat incoming content as evidence, not authority

The model processes untrusted material: emails, PDFs, websites and retrieved documents. Prompt injection, an instruction hidden in material the agent reads, can persuade it to misuse the tools if the application grants excessive functionality, permissions or autonomy. System instructions are not a security boundary that substitutes for identity and authorization.

## List the valuable data and permitted actions

Begin by listing the valuable information the system can reach and the actions it can request. Identify what each tool reads or changes, whose identity is used, and which operations are reversible. Restrict the allowed inputs and outputs, enforce fixed permission and business rules outside the model, and require approval for high-consequence actions. Build logs that allow an investigator to reconstruct what happened.

## Put a malicious instruction in the supplier document

Inject malicious text into a sample supplier PDF requesting the agent to send data to an unrelated endpoint. Confirm the tool layer rejects any unsupported operation. Then try duplicate requests, role changes and failed writes. A successful defense cannot rely solely on the model saying no.

## Match controls to what the agent can change

Read-only agents and low-impact suggestions may need lighter controls. An agent that changes financial terms or accesses sensitive records needs a much stronger architecture. Autonomy is not a binary product feature.

## Review authority whenever a tool is added

Before release, document the allowed tools, whose permissions each action uses, how credentials stay protected and when human approval is required. Include how repeated requests avoid duplicate changes (idempotency), the action history, incident response and tests of forbidden behavior. Review the controls whenever tools are added.

### References and context

[OWASP Top 10 for LLM Applications](<https://owasp.org/www-project-top-10-for-large-language-model-applications/>); [NIST AI RMF Generative AI Profile](<https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence>).

## Sources

- [Generative AI Profile for AI RMF](<https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence>) — NIST. Reference governance and risk design principles for gen AI. Risk management reference for generative AI Limit: This is a risk-management reference, not a certification or evidence that Marmation has been audited or complies with it.
- [Top 10 for LLM Applications](<https://owasp.org/www-project-top-10-for-large-language-model-applications/>) — OWASP. LLM risks such as prompt injection, excessive agency and data disclosures. Threat categories for LLM systems Limit: This is an engineering security reference, not evidence of an independent audit of Marmation or another system.

## Related reading

- [From working prototype to dependable system](https://marmation.com/review/engineering/)
- [Design the action boundary before designing the agent.](https://marmation.com/review/designing-an-agent-with-bounded-actions/)
- [Engineering autonomy is a control design problem.](https://marmation.com/review/engineering-autonomy-is-a-control-design/)