The case for, and against, model-provider abstraction.

A team wants to make its AI product model-agnostic from the first week. It designs a wrapper that exposes the same interface for every model, tool format and output capability. The abstraction is elegant on paper but eliminates useful features and complicates testing.

The decision behind the technology

Different models can have different structured-output behavior, context limits, tool calling, cost, regional availability and latency. Treating them as interchangeable at every level can either weaken the application or create a costly lowest-common-denominator platform.

A better way to frame the work

Abstract the parts of inference that the product truly needs to switch: request/response contracts, credentials, error handling, logging and evaluation. Isolate provider-specific capabilities explicitly rather than pretending they do not exist. Use a representative benchmark to determine which portability matters.

Put the idea under realistic pressure

A knowledge assistant might be portable across two approved model services with little UI change. A sophisticated streaming agent with provider-specific tool behavior may require more adaptation. Document the switching cost rather than promising one-click migration.

When the recommendation changes

In a single-use application with stable requirements, direct integration may be simpler and safer. For products with sovereignty, pricing or resilience constraints, a defined provider boundary can be valuable.

What to decide before moving forward

Before building an abstraction layer, identify the real risk being insured: cost, continuity, region, legal terms or model quality. Keep architecture proportional to that risk.

References and context

McKinsey 2026 AI survey; NIST AI RMF.