Cloud Infrastructure for AI and Machine-Learning Workloads

AI workload is a broad phrase covering activities with genuinely different infrastructure profiles. Separating them makes the planning tractable.

Three distinct activities

Training from scratch

The most demanding and, for most organisations, the least relevant. It needs large amounts of accelerator capacity for extended periods, fast interconnects between nodes, and high-throughput storage. Accelerator quotas are usually zero by default and require an explicit request.

Fine-tuning

Adapting an existing model. Far smaller: often a single accelerator for hours rather than clusters for weeks. This is within reach of ordinary budgets, and it is bursty, which suits hourly billing well.

Inference

Running a trained model to produce results. Where most production AI spend actually goes, and the profile is quite different: continuous rather than bursty, latency-sensitive, and scaling with user traffic rather than dataset size.

The managed-service shift

The most significant practical change is that many teams no longer provision accelerators at all. Managed model services expose models over an API, billed per request or per token, with the provider handling capacity.

That changes the infrastructure question from what hardware do I need to which service is enabled on my account and what request rate am I permitted. This is precisely what AWS Bedrock-enabled configurations describe, and why the request-rate figure appears alongside them. See reading AWS configurations.

What still needs ordinary infrastructure

Even with managed models, the surrounding system is conventional:

• Application servers handling requests and orchestration.
• A database for application state and conversation history.
• Object storage for documents, embeddings sources and artefacts.
• A queue for work that cannot be done inside a request.
• Logging and cost monitoring, which matter more than usual because per-token billing scales with usage in ways fixed servers do not.

Most of the engineering is ordinary cloud engineering.

Cost behaviour to watch

Two patterns deserve attention. Accelerator instances are expensive per hour and easy to leave running after a job finishes. And per-token billing scales directly with traffic, so a retry loop or an unbounded context can multiply cost without any visible change in behaviour.

Budget alerts matter more here than in conventional workloads. See budget alerts that actually fire.

Choosing a platform

If you intend to use managed model services, service availability and enabled state on the account are the deciding factors rather than raw compute. AWS and Google Cloud are the usual candidates. For the surrounding application infrastructure, any competent provider will do.