The Data & Knowledge Foundation
A cross-cutting capability, not a lifecycle stage. Five pillars for data that's actually ready to serve AI — sourcing & ingestion, quality & curation, governance & rights, retrieval & serving, and knowledge structure — plus an 8-point AI-ready data test. Poor data quality is Gartner's #2 reason GenAI projects stall; your data is also the one advantage a competitor can't buy.
By Tom Ward, Enterprise Architect — Cloud & AI
The five data pillars
Sourcing & Ingestion
“Getting the right data in”Every AI capability is downstream of the data that feeds it. Map where the right data lives, build the connectors and pipelines to reach it, and be honest about coverage gaps before they become production surprises. Batch vs. streaming is set by the freshness the use case needs, not by fashion.
Quality & Curation
“Garbage in, hallucinations out”Gartner puts poor data quality near the top of its GenAI failure list — because bad data doesn't announce itself, it produces confident, plausible, wrong answers. Quality is a continuous SLA, not a one-time cleanse, and for GenAI the chunking and labeling that prepare data for retrieval are quality decisions in their own right.
Governance & Rights
“Who may use what, lawfully”Where the Data and Governance capabilities meet. Retrieval must enforce the same entitlements as the source systems — if a user can't open the document, the model must not surface its contents. Add lineage (to explain and audit a decision) and honest usage-rights checks (are you legally permitted to use this data for AI at all?).
Retrieval & Serving
“Getting it to the model”Good, governed data still has to reach the model fast and relevant. Embeddings, vector and hybrid search, re-ranking, and serving infrastructure that returns the right context within the experience's latency budget. Retrieval quality — more than model choice — often decides whether an answer is right, and a stale index quietly rots every answer.
Knowledge Structure
“Meaning, not just bytes”The most mature foundations capture meaning, not just data — metadata, ontologies, and knowledge graphs that turn a pile of documents into a connected model of your business. The most-skipped pillar, because it's hardest and its payoff is longest-term, but it's the deepest moat: a semantic layer every future use case can draw on.
The AI-ready data test
“Is our data ready?” is the wrong question — unanswerable at scale, and it stalls everything. Ask it per use case. For a specific initiative, data is AI-ready only when all of these hold; each “no” is a scoped, fundable task.
- Reachable — the data this use case needs is accessible through a real, supported path, not a one-off export.
- Sufficient — coverage and history are complete enough that the model isn't guessing in the gaps.
- Trustworthy — quality is measured and monitored, not assumed from a clean demo sample.
- Governed — retrieval enforces source-system entitlements at query time, with lineage captured.
- Lawful — you have the usage rights and privacy clearance to use this data for this AI purpose.
- Fresh — the refresh cadence matches how fast the underlying data actually changes.
- Retrievable — the right context comes back fast enough, and retrieval quality is evaluated on its own.
- Meaningful — where the use case needs it, a semantic layer gives the data context, not just similarity.
Going deeper
Sources worth your time, roughly in the order a program team should read them.
- DAMA-DMBOK — Data Management Body of KnowledgeThe reference framework for data management disciplines — governance, quality, metadata, and more.
- Microsoft — Azure Data Architecture GuideConcrete patterns for data platforms, pipelines, and stores — translates well beyond Azure.
- Databricks — The Data LakehouseThe lakehouse pattern for unifying analytics and AI data on one governed platform.
- Google Cloud — Retrieval-Augmented GenerationA clear primer on RAG — the mechanism that gets your governed data to the model.
- Martin Fowler — Data Mesh PrinciplesA vendor-neutral take on data ownership and domain-oriented architecture at scale.
- Data Observability — an introductionHow to monitor data quality and freshness continuously, the way you monitor uptime.
- NIST AI Risk Management FrameworkThe governance vocabulary for the data-rights and privacy questions in Pillar 3.
Next parts ship over the coming weeks.
Get the next part by email