How data architecture works at LLM companies is a question many internet data practitioners are curious about. On the surface the keywords look similar (large-scale data processing, pipelines, quality governance), but the actual practices and underlying logic differ a lot.

I’ve had some conversations with LLM practitioners recently and picked up a few things. This post breaks it down from a technical angle: how the training side and the application side of LLMs differ, at the core, from the data architecture of internet companies.

The Two Main Lines of Internet Data Architecture

Analytics: Data Warehouse + BI

The core is a business metrics system plus multi-layer warehouse modeling. Data flows from ODS (raw layer) through DWD (detail layer) and DWS (aggregate layer) to ADS (application layer), building dimensional models (star/snowflake) to support reporting and self-service analysis. The architectural concerns are metric consistency, standardized metric definitions, query performance, and access control. Organizationally there are usually dedicated warehouse and BI teams.

Algorithms: Feature Engineering + Model Serving

The core is offline feature computation plus online feature serving. User behavior produces event streams, ETL pipelines clean and aggregate them into a feature store, and at prediction time features are read from Redis/HBase, assembled into vectors, and fed to the model. Features are structured, predefined, and fixed-dimensional. The architectural concerns are feature consistency (training vs. serving), latency (Lambda/Kappa), feature lineage, and cold start.

These two lines share the underlying storage and compute, but they serve different consumers, with different quality standards and optimization goals. The data platform team usually has to guarantee report accuracy and feature freshness at the same time, while keeping the two isolated from each other.

Training Side: A Paradigm Shift in Data Pipelines

Training data splits into two halves, pretraining and post-training. Post-training data (SFT dataset construction, RLHF preference data labeling and QC, synthetic data generation) is more of a hybrid of data engineering, annotation operations, and research, and it diverges from internet data work even further. This post focuses on the pretraining pipeline.

An LLM’s pretraining pipeline is corpus-driven batch iteration. Collect from massive heterogeneous sources (CommonCrawl, code repositories, books, synthetic data), run deduplication, quality filtering, harmful content filtering (toxicity detection), and PII removal, assemble dataset snapshots, and feed them into training mixed by ratio (the data mix).

Key Differences from Internet Data Pipelines

Data acquisition: the challenge shifts from “integration” to “compliance and quality”

An internet company’s data mostly lives in its own product’s event tracking. The data team’s job is to onboard logs from each business line: is the protocol right, is the format consistent, is the link stable. An LLM company’s data comes from outside: crawl the web, negotiate licensing with publishers, design rules to filter out low-quality content, use models to evaluate corpus quality.

On top of that, the data mix itself is an optimization target: DoReMi trains a small proxy model to learn the optimal weight for each source, and the Llama 3 technical report describes its mixture tuning process in detail. Large-scale corpus quality evaluation becomes core work for the data team. Licensing negotiation itself belongs to legal and BD, but the data team supplies the evidence (how much is this corpus worth to model quality) and maintains the licensing status and provenance records for every source. None of this work has a counterpart on an internet data team.

What schema constrains has changed

Internet data pipelines lean heavily on predefined schemas (Hive tables, Protobuf); fields, types, and definitions are agreed on at write time. Pretraining corpus content is unstructured, but the pipeline itself is highly structured: documents are normalized into a standard format (jsonl/parquet, text plus metadata) offline, and quality filtering, deduplication, and validation all happen offline. Many labs also pre-tokenize and pack sequences into fixed-length binary files, so training only does sequential reads.

The difference is not whether there is a schema. It’s that what the schema constrains has moved from business fields to document format plus metadata, and what gets validated has moved from field-level definitions to corpus quality.

What gets versioned has changed

Internet data pipelines version ETL code and table schemas; rolling back the platform means rolling back code. Model training pipelines version dataset snapshots, data mix ratios, and deduplication strategies. The dataset itself becomes a first-class citizen.

When a training run goes wrong, you need to trace which dataset version it used, how it was sampled, and which dedup strategy was applied. Before releasing a model, you need to prove the training corpus excluded the eval sets (decontamination), or the benchmark scores can’t be trusted. Internet data platforms have lineage too (table-level, column-level), but the object and granularity differ: there it tracks how tables and columns are derived; here you need immutable snapshots plus experiment-level provenance.

The compute pattern shifts from streaming/scheduled batch to long-cycle bulk

Internet ETL runs on hourly/daily schedules with incremental updates. Data preparation for model training is a weeks-to-months job that processes petabytes of corpus in one pass, and the snapshot is frozen once it’s done. Training usually makes only one epoch over the corpus (high-quality subsets get upsampled a few times).

The hard parts are elsewhere: petabyte-scale data needs a global shuffle, and the data loader has to be deterministic. After an interruption, training must resume from exactly the same position in the data, or the experiment isn’t reproducible. Storage requirements differ too: internet systems emphasize high-concurrency writes and queries; model training emphasizes sequential read throughput.

Quality evaluation shifts from business rules to model-driven

Internet data quality is business rules (null rates, enum ranges, logical consistency) plus manual spot checks. Corpus quality can’t be specified that way. “Is this a good document” can’t be written as a validation rule; only a model can judge it: compute perplexity (how fluent a language model finds the text), score toxicity, or train a dedicated small classifier to grade documents. The heaviest tool is ablation: remove one category of data from the corpus, retrain a small model, compare the results, and verify in reverse whether that data actually contributes. The quality standard itself is a research question, not an engineering spec.

Application Side: From Structured Features to Dynamic Prompt Context

The internet algorithm side was covered above: features are structured, predefined, fixed-dimensional.

LLM applications use data differently: retrieve relevant documents at runtime and assemble them into the prompt context, which may include retrieved knowledge, few-shot examples, tool call results, and multi-turn conversation history. Context is unstructured, dynamically generated, and constrained by a token budget. Adding a new knowledge source only requires embedding and indexing; the model doesn’t change.

The architectural differences cascade from these two usage patterns.

Storage moves from row-oriented key-value to vector index plus raw text. Internet features live in Redis, keyed by user_id or item_id, with feature vectors as values. Model applications need a vector database for semantic retrieval, while keeping the raw text for prompt assembly. Index and storage are separate: when the embedding model is upgraded, you only rebuild the index; the raw text stays.

The freshness trade-off is different. Internet feature stores chase second-level updates because user behavior immediately affects the next recommendation. A model application’s knowledge base can tolerate hour-level or even day-level index update delays, but the bar for retrieval quality is very high: retrieve the wrong documents, or documents that contradict each other, and the whole inference goes sideways.

The quality standard moves from data metrics to end-to-end inference quality. Internet feature engineering looks at feature coverage and missing rates, metrics about the data itself. RAG retrieval has its own metrics too, like recall@k (whether the relevant documents are in the top k results) and MRR (how high the relevant documents rank), but these are only for debugging, not acceptance. Acceptance is the accuracy and relevance of the final answer. That means the data team gets deeply involved in prompt tuning and eval design, instead of just delivering data.

The hard part of governance has changed. Internet data governance is user profile anonymization, access control, regulatory compliance. All of that still has to happen in model applications, and it gets harder: access control used to stop at tables and columns; now retrieval finds content by semantics, so permissions have to answer “which documents must this user’s query never retrieve,” and one document with misconfigured permissions can end up assembled into someone else’s prompt at any time. There is also a new class of threat: malicious instructions can be written directly in user input, or hidden in knowledge base documents and stitched into the prompt via retrieval, steering the model into saying things it shouldn’t. That is prompt injection.

What Data Teams Do Under Each Architecture

Data work on the training side

The training side needs: large-scale data processing engineering (dedup and sampling at petabyte scale), data quality engineering (designing filter pipelines, using models to evaluate corpus quality), and compliance processes (licensing and provenance management for data sources, harmful content review). These partially overlap with the foundational capabilities of an internet data platform.

The focus of domain understanding changes: from user behavior and business metrics to how corpus quality affects model capability. Which data improves reasoning, which data introduces bias.

The new parts of the stack include dataset version management, large-scale deduplication, synthetic data generation, and data mix optimization.

Data work on the application side

Retrieval augmentation and prompt context management: how to design embedding and indexing strategies, how to do retrieval and ranking, how to assemble context within a token budget, how to give agents effective memory.

The stack is completely different from internet warehousing and feature engineering: vector databases, embedding models, semantic routing, and prompt caching are the new infrastructure.

The quality standard was covered above: end-to-end inference quality, not metrics about the data itself. In day-to-day terms, prompt tuning and eval design become part of the data job.

Organizational differences

An internet data platform is a shared platform serving the entire company; the data team is relatively independent and organized by technical domain (warehousing, engineering, data science).

At LLM companies, data teams are usually embedded directly in model development (training side) or product (application side), not run as an independent platform. The data lead on the training side takes part in model scaling decisions (data scaling laws); the data lead on the application side takes part in the product’s prompt design and eval framework.

The two architectures also prove a data team’s value differently. Internet: platform scale and service stability. LLMs: direct contribution to model quality.