Assessing the AI Stack from Infrastructure to Interface
The finale, and the map back through the series: a five-layer reference architecture — compute & networking, data platform, model layer, orchestration, experience — scored on a four-level maturity scale, with security, cost, observability, and portability assessed at every layer. Know what you have, then decide what to build.
By Tom Ward, Enterprise Architect — Cloud & AI
The five stack layers
Compute & Networking
“Can it run at scale?”The foundation everything stands on, and the most expensive layer to reverse. AI workloads have an unusual appetite for accelerators, memory bandwidth, and east-west throughput — a stack that serves web traffic beautifully can buckle under inference. Assess for peak inference and training load at a defensible cost, not for the demo.
Data Platform
“Is data ready to serve?”Where the differentiator actually lives — and usually the quietest immaturity. Serving AI demands low-latency retrieval, vector and feature stores, and fresh pipelines, not just a good quarterly report. Governance belongs in this layer, enforced where it propagates upward, not bolted on above where it can be bypassed.
Model Layer
“Which models, how served?”The layer everyone thinks of first and assesses least rigorously. Maturity shows up as optionality: a gateway and registry that give you a catalog, cost attribution, routing, and the ability to swap model or provider as a config change, not a project. Three teams calling three APIs directly isn't a model layer — it's future technical debt.
Orchestration
“How do the pieces coordinate?”Where a pile of components becomes a system — retrieval, agents, tools, and context management — and where the agentic capabilities from Part 4 actually run. The newest, least-standardized layer. Assess for repeatability, not cleverness: shared retrieval and a common, observable agent framework beat brilliant one-off plumbing every team reinvents.
Experience
“How do users meet it?”The top of the stack and the only layer users see. AI embedded where work already happens gets used; AI in a separate portal gets abandoned. Assess for reach into real workflows, a clean API surface for composing AI into products, and the latency and trust the workflow demands. A great stack with a poor experience layer delivers no value.
Score each layer 0–3, then read the gaps
Rate every layer on the maturity scale — 0 Absent, 1 Emerging, 2 Standardized, 3 Optimized — and weigh it against the four cross-cutting concerns. The total isn't a grade; your lowest layers are your roadmap.
- Compute & Networking has capacity headroom for peak inference, not just the average — and a deliberate cloud/on-prem/hybrid split.
- The Data Platform serves low-latency, governed, fresh data — with governance enforced in-platform, not above it.
- The Model Layer reaches applications through a gateway and registry, and you can swap a model or provider as configuration.
- Orchestration offers shared retrieval and a standard, observable agent/tool framework — not per-team improvisation.
- The Experience layer embeds AI in real workflows with the right latency, not a stranded destination app.
- Security, Cost/FinOps, Observability, and Portability are each assessed at every layer, not just one.
Going deeper
Sources worth your time, roughly in the order a program team should read them.
- a16z — Emerging Architectures for LLM ApplicationsThe widely-cited reference architecture for the LLM app stack — a strong baseline to map your own layers against.
- Martin Fowler — Emerging Patterns in Building GenAIVendor-neutral engineering patterns for the orchestration and experience layers, from a trusted architecture source.
- Azure Well-Architected — AI WorkloadsProduction design guidance across the stack: reliability, cost, security, and operations. Translates beyond Azure.
- AWS Well-Architected — Generative AI LensAWS's counterpart lens: lifecycle, cost, and operational best practices for GenAI workloads.
- Google Cloud — AI & ML ArchitectureReference architectures for the data, model, and serving layers — the third hyperscaler view for triangulation.
- CNCF — Cloud Native AI WhitepaperThe open, vendor-neutral view of the infrastructure and platform layers — Kubernetes-era AI foundations.
- NIST AI Risk Management FrameworkThe governance vocabulary for the cross-cutting concerns — map your stack's controls to it.
Next parts ship over the coming weeks.
Get the next part by email