Where Does Your AI Actually Run?
Managed APIs, managed private infrastructure, rented GPUs, colocation, and on-premise systems balance control, cost, and responsibility differently. Here is how the main options compare.
Using AI can feel almost weightless. You open a chat, type a question, and receive an answer seconds later. But every answer has to be computed somewhere. A model is running on physical hardware, in a particular jurisdiction, under someone’s operational control, with specific rules for how data is processed and retained.
For casual use, those details may not matter much. For an organisation connecting AI to internal documents, customer records, product data, or regulated information, they matter enormously. The question is no longer only which model to use. It is also where that model should run.
Managed services offer remarkable capability with little infrastructure work. Customer-controlled environments provide greater isolation and configurability, but require more money, expertise, and operational responsibility. The right choice depends on the data, the workload, and what the organisation is prepared to manage.
First, separate the application from the model
An AI application and the model it uses do not necessarily run in the same place.
A company might self-host its chat interface and database while sending every prompt to a public model API. From the user’s perspective, the application is private. From the perspective of inference, the data still leaves the company’s environment.
The reverse is also possible. A managed application can connect to a model endpoint reachable through the customer’s private network. The interface is operated as a service, while inference may still run on provider-operated compute that the organisation can reach privately, or, in other setups, on compute the organisation controls.
This is why “self-hosted AI” can mean several things. Hosting the application yourself does not automatically mean hosting the model yourself. When evaluating privacy, compliance, and control, the important question is where the data goes when the model produces an answer.
Select different combinations below to see how the application, inference endpoint, and physical hardware can sit under different operational boundaries.
Interactive prompt journey
Where does the request actually go?
Change the application and inference environments independently. The interface location does not determine where the model runs.
Application hosting
Inference environment
Application
Self-hosted AI application
Your organisation · Your environment
Inference endpoint
Provider-operated model API
Model provider · Provider environment
Physical compute
Shared provider compute
Hardware controlled by model provider
The provider operates the model endpoint and compute. Enterprise terms can strengthen retention, regional processing, and audit commitments, but do not change who operates the infrastructure.
Managed AI: capability on demand
The most familiar option is a managed provider. You send a request to an API, the provider runs the model, and the response comes back. The provider manages the hardware, model serving, scaling, updates, and availability.
This model is one reason AI adoption has moved so quickly. Organisations can access highly capable models without purchasing a GPU or building a specialised infrastructure team.
The advantages are compelling:
- Fast setup and very little operational overhead
- Immediate access to capable and frequently updated models
- Usage-based costs that work well for experimentation and variable demand
- Global infrastructure and mature developer tooling
The trade-off is control. The provider determines how inference is operated, so customers must understand the data-processing terms, retention settings, supported regions, security controls, and model lifecycle. Costs can also become less predictable when long conversations, large documents, or automated agents generate substantial token volumes.
Enterprise plans are a governed version of this same deployment model, not a separate kind of infrastructure. They may strengthen retention, training, regional processing, access, audit, support, and liability terms. Those changes affect the commercial and data-protection relationship; they do not change who operates the GPUs. Inference still runs in infrastructure operated by another company.
Managed AI fits public information, low-risk workloads, prototypes, and teams that value speed over infrastructure control. It can also support sensitive production workloads when the provider’s architecture, terms, and controls meet the organisation’s requirements. Convenience is a benefit, not a substitute for data governance.
Managed private infrastructure: greater isolation, shared responsibility
The next option moves inference into a more isolated provider-managed environment. This might be a service tied to a particular tenant or cloud project, a privately networked endpoint, or dedicated model-serving capacity.
Services such as Azure OpenAI, Amazon Bedrock, and Google Vertex AI illustrate parts of this model. The exact architecture and responsibilities differ, so the label “private” should never be accepted without examining what it means in practice. In particular, separate:
- Network isolation: private connectivity to an endpoint, without assuming the compute sits inside the customer’s environment
- Tenant isolation: logical separation within a shared cloud service
- Dedicated capacity: reserved hardware or serving capacity for one customer
- Customer-operated inference: the customer runs the model-serving stack themselves (the next section)
A privately networked managed endpoint can improve data-path control while the provider still operates the model and compute. Specialist GPU platforms can provide similar arrangements: a shared serverless worker remains provider-managed; dedicated capacity with private networking offers stronger isolation. The product label matters less than the actual tenancy, networking, operational control, and data-processing terms.
This model can combine managed access with familiar cloud governance:
- Private networking and tighter control over data paths
- Regional deployment and data-residency options
- Integration with existing identity, logging, and security systems
- Less infrastructure work than operating model servers directly
The trade-offs are increased complexity and potential cloud lock-in. A private endpoint can still be undermined by permissive access policies, incomplete logging, or misunderstood retention settings. Model availability may also be narrower or arrive later than through a provider’s direct public API.
Managed private infrastructure fits organisations already committed to a major cloud, or those that need AI close to existing data and security controls. It offers more isolation than a general-purpose public API, but security still depends on shared responsibility between customer and provider.
Customer-hosted AI: rented hardware, customer-controlled deployment
Organisations can also deploy open-weight or licensed models themselves without buying physical hardware. They rent GPUs from a public cloud or specialist provider and operate the inference stack on top of them.
The customer might use a GPU virtual machine, dedicated bare-metal server, or Kubernetes cluster. The defining feature is not ownership of the machine. It is that the provider supplies infrastructure while the customer operates the model-serving environment. Tools such as vLLM and TensorRT-LLM can turn that infrastructure into an API for internal applications. The hardware can be rented while the inference deployment remains customer-controlled.
Customer-hosted inference offers control over the model, runtime, configuration, and update schedule; the ability to deploy open or customised models; greater influence over logging, retention, networking, and access; and potentially attractive economics for sustained, well-utilised workloads.
It also transfers substantial responsibility to the customer. Someone must secure the environment, manage model versions, monitor performance, handle failures, plan capacity, patch dependencies, and keep expensive hardware productively occupied.
On-demand machines suit experiments and variable demand. Reserved or dedicated servers offer more predictable capacity for steady production traffic, but continue to cost money when underused. Spot capacity can be cheaper, although interruptions make it unsuitable for some real-time workloads.
Customer-hosted AI is attractive when an organisation needs custom models, tighter technical control, or predictable high-volume inference, and less so when the infrastructure burden distracts from the problem the AI was meant to solve.
Customer-owned infrastructure: maximum locality and responsibility
At the furthest customer-controlled end, inference runs on hardware owned or fully controlled by the organisation.
With colocation, the organisation places its hardware in a third-party data centre. The facility provides rack space, electricity, cooling, physical security, and connectivity, while the organisation remains responsible for the servers and inference stack. Colocation is not the same as renting the facility’s servers: when the provider supplies the machine as a service, that is dedicated or bare-metal hosting. In colocation, the organisation controls the physical equipment even though it is housed elsewhere.
With on-premise or isolated deployment, the hardware sits in a company data centre, research environment, factory network, or air-gapped system. This can keep sensitive data inside a defined physical boundary, support operation without external connectivity, and address specialised sovereignty or regulatory requirements.
Smaller models can also run directly on laptops, phones, vehicles, and industrial equipment. This edge or on-device inference offers offline operation and strong data locality, but local memory, power, and compute limit which models can run effectively.
Locality should not be confused with speed. Large cloud providers invest heavily in specialised inference infrastructure, and the time spent generating tokens usually dominates the network round-trip. A typical on-premise deployment will not match that performance without a comparable capital outlay.
These benefits come with the highest commitment. Hardware must be purchased, installed, maintained, and eventually replaced. Owned capacity cannot scale with an API call, and unused equipment still carries depreciation, energy, and maintenance costs. Customer-owned and edge infrastructure make the most sense when locality is a requirement, not merely a preference.
From paying for usage to paying for capacity
Deployment changes how an organisation spends money on AI. Managed APIs are usually an operating expense (OPEX): the organisation pays for usage without owning the underlying hardware. A €500 monthly inference bill is a simple example. Rented GPUs are also generally OPEX, but reserved machines shift the decision from paying for individual requests towards paying for available capacity.
Buying hardware introduces capital expenditure (CAPEX), for example a GPU server purchased upfront, alongside continuing costs for electricity, cooling, networking, maintenance, and staff. The practical question is simpler than the accounting terminology: are you paying for the inference you use, or for capacity you must keep productive?
Usage-based services reduce upfront commitment and work well when demand is uncertain. Owned or reserved capacity can lower the unit cost of stable, high-volume workloads, but only when that capacity is used effectively. An idle API makes no requests. An idle GPU still costs money.
This is where deployment strategy connects directly to token economics. The cheapest price per token is not necessarily the cheapest system once unused capacity, engineering time, maintenance, and resilience are included.
The spectrum is not a security ranking
A carefully governed enterprise API may be safer than a poorly maintained private deployment. A private cloud environment with excessive permissions may expose more data than a public service configured with strict retention controls. An on-premise server that is rarely patched does not become secure simply because it sits behind a company firewall.
Deployment changes who is responsible for managing risk; it does not remove risk. Moving towards customer-controlled infrastructure generally increases control and isolation, but also increases the security, reliability, and operational work the organisation must perform. Control is valuable only when matched by the capability to exercise it.
Most organisations will use more than one option
The most realistic AI architecture is not a universal deployment. It is a portfolio. Public information and low-risk productivity tasks may be handled efficiently through managed APIs. Customer data might require enterprise terms or a privately networked endpoint. Highly confidential intellectual property may need customer-hosted inference. A factory or remote site may require an on-premise or edge model that works offline.
These choices can even exist within the same application. What matters is that the system understands the difference between workloads and routes each one to an appropriate environment. That avoids two expensive extremes: sending every task through the most restricted infrastructure, or treating every piece of data as suitable for a general-purpose public service. It also requires visibility: which model handled a request, where inference took place, and what rules determined the route.
Ask these questions
Before choosing an environment for a workload, ask:
- How sensitive is the data, and which legal or residency rules apply?
- Does the workload need a frontier managed model, an open-weight model, or a custom one?
- How steady is demand: sporadic experiments, or sustained high-volume inference?
- Does the organisation have the people and processes to operate and secure the stack?
- Must the system work offline, or keep data inside a defined physical boundary?
- What does the full cost look like once unused capacity, engineering time, and resilience are included?
The takeaway
Where AI runs shapes what an organisation can safely and economically do with it. Managed options maximise accessibility; greater isolation and ownership increase control and responsibility. Deployment is a spectrum of trade-offs, not a security ranking. The mature question is not “Which deployment model should we choose?” It is “Which environment is appropriate for this workload?”
This article is part of a foundational series on AI literacy and adoption. You might also want to read AI Is Here to Stay, Why Multi-Provider AI Matters, and Token Economics: The Unit of AI Compute.