An Intro to LLM Training at Scale: How Modern AI Systems Stay Fast, Big, and Secure
- Ryan Murphy
- Jul 11
- 10 min read
Updated: Jul 14
A tale as old as time: modern AI models are too large, complex, and resource-intensive for a single chip to handle. Training them is not like running an application on a single machine. It's a distributed systems problem, closer to coordinating a warehouse full of machines that all have to stay in lockstep.
I did a deep dive into this topic from a Security Engineer's point of view back in April 2026, when I was certain AI was going to take my job. Since then, my view on my job security has done a complete 180, thanks to a string of high-profile AI security incidents (see LiteLLM and others). Not to mention the looming Mythos software-apocalypse (!).
This post aims to break down how coordinating and securing model training works in simple terms. This is a complex topic, but my goal is to provide a high-level overview for engineers who have limited experience with how LLMs are trained and secured under the hood. You do not need a machine learning background to benefit from this post. If you understand servers, networks, and the basics of distributed systems, you have enough to follow along. All terminology, examples, and descriptions are intentionally minimal, meant to demonstrate concepts rather than adhere to exact technical specifications.
Concepts
Before we begin, it's worth defining some important concepts and terminology used throughout this post.
Model: At a high level, software that learns from data and gets smarter over time. When you hear about LLMs (Large Language Models), this is the core concept.
Training Data: The dataset used to train a model by exposing it to examples, from which it learns patterns, relationships, and behaviors during the training process.
AI Accelerator Chip (GPU/TPU): A specialized computer chip built to train and run AI models. Unlike a general-purpose CPU, an accelerator is purpose-built for the matrix math that powers modern AI, making it significantly more efficient for these workloads. GPUs (primarily from NVIDIA) are the most widely used; TPUs are Google's proprietary alternative, available only through Google Cloud.
Roofline: A framework that identifies the hardware limits on software performance consisting of compute, memory, and bandwidth. When an algorithm maxes out one of these limits, it becomes "bound" by that resource.
Strong Scaling: A metric for how efficiently a training job scales as more GPUs are added. Ideally, as you add GPUs, the speed at which your model is processed scales gracefully.
Shard: A single piece of a larger whole. When data, a model, or a computation is too large to fit in one place, it is split into smaller pieces called shards. We use this term loosely for consistency.
The Problem: Training large models is hard
Models are massive. Frontier models can have trillions of parameters (learned values), far exceeding the capacity of a single chip.
Datasets are enormous. Every example has to pass through the model during training. Even if the model fits on one chip, streaming the entire dataset through it would take far too long.
Training is time-sensitive. For both cost and competitive reasons, models can't take an excessive amount of time to train, so the obvious goal is to add chips and expect the training job to finish sooner.
Security is easy to under-prioritize. When the hard part is getting the job to run at all, security tends to arrive last. Models are amongst the most valuable IP a company owns, yet training infrastructure may not be secured beyond the perimeter.
The first three issues all hit the same wall, the physical limits of the hardware (the roofline). Max out any one of these and you're bound by it.
Compute is how much math the chip can do per second.
Memory is how much data the chip can hold at once. This is the limit that large models run into first.
Bandwidth is how fast data moves, on-chip and between chips.
The most common failure at scale is being communication bound. At some point, adding chips increases communication overhead instead of optimizing for speed because the chips spend more time talking to each other than doing actual math.
The Solution: Achieving Strong Scaling through Parallelism
In order to achieve strong scaling, we attempt to run model training computation in parallel. There are several ways to achieve this, each with its own benefits and tradeoffs.
Data Parallelism: Replicate the entire model across multiple chips; each chip processes a different slice of the training data simultaneously.
Tensor Parallelism: Shard the model across multiple chips, with each chip handling a part of the same computation.
Pipeline Parallelism: Model layers are split into sequential stages, with each stage running on a different chip (i.e. assembly line).
3D Parallelism: The production standard; combines all three techniques simultaneously.
A fourth technique, Expert Parallelism, scales a different kind of model and is out of scope here; we will cover it briefly at the end.
Data Parallelism: Scaling the data
Concept
Data Parallelism is generally considered the simplest approach to parallel model training. The entire model is copied across multiple chips, where each chip processes a different slice of training data simultaneously. Every chip does the same work on its own slice, and they periodically sync up (combining gradients) so copies stay consistent.
The limitation shows up with large models. Every chip holds a full copy, so the model has to fit in a single chip's memory; if it doesn't, data parallelism can't help. It scales throughput, not model size, which makes it the right tool when your bottleneck is the volume of training data rather than the size of the model itself.
Implementation: Proof of concept
A Kubernetes Job fans out N identical workers at once; each worker is a full model replica pinned to a single GPU. Each worker reads a different shard of a shared, read-only dataset, and a headless Service lets the workers find each other for the periodic sync.
#####################################################
# Basic proof-of-concept; not for production use
#####################################################
apiVersion: batch/v1
kind: Job
metadata:
name: dp-training
spec:
parallelism: 4 # 4 workers, all replicas of the full model
completions: 4 # require 4 completions for job completion
completionMode: Indexed # each worker gets a stable index, its "rank"
template:
spec:
subdomain: dp-workers # must match the Service name below
containers:
- name: worker
image: registry.internal/trainer:1.0
resources:
limits:
nvidia.com/gpu: 1 # one accelerator per worker
volumeMounts:
- name: dataset # shared dataset, never modified
mountPath: /data
readOnly: true
volumes:
- name: dataset
persistentVolumeClaim:
claimName: training-data
restartPolicy: Never
---
apiVersion: v1
kind: Service
metadata:
name: dp-workers
spec:
clusterIP: None # headless: stable network group for syncing
selector:
job-name: dp-trainingSecurity: Key Considerations
Data parallelism's security risks center on its defining trait, many identical workers that all start at once and share the same data.
Data exfiltration and integrity. Models and their training sets are amongst the most valuable IP a company owns, which makes this one of the top risks.
Grant workers read-only access to the dataset, scoped to only the shards they need.
Encrypt data in transit between workers and at rest in shared storage, so intercepted syncs and stolen volumes are useless.
Apply strict egress network policies so worker pods can only reach each other and required internal services, never arbitrary external destinations.
Thundering herd at startup. All workers start simultaneously and load the full model, creating a surge of requests to secret stores, identity providers, and shared storage that can overwhelm backend systems.
Ensure shared services scale horizontally or distribute load effectively.
Use OIDC (or SPIFFE/SPIRE) to issue short-lived credentials upon worker creation.
Rogue workers. Anyone who can schedule a pod with matching labels gets added to the discovery group and can poison the synchronization.
Enforce security controls throughout the deployment pipeline and within the Kubernetes cluster to restrict pod creation in the training namespace.
Enforce pod-to-pod mTLS (via SPIFFE/SPIRE) so workers can reject an impostor outright.
Tensor Parallelism: Scaling the model
Concept
When the model is too large to compute on one chip, you need to split the math itself. In tensor parallelism, each chip holds only the slice of the calculation it is responsible for, while the chips continuously sync, or combine, their partial results into the full answer. Instead of replicating the model on each chip, you shard it.
Tensor parallelism comes at the cost of bandwidth, as you add more chips, you increase the risk of the system becoming communication bound. Since tensor parallelism relies heavily on synchronization, latency becomes the bottleneck. This is why it demands the fastest interconnects available and why the chips usually have to be co-located on the same machine or within the same high-bandwidth group. When done correctly, it allows you to pool the memory of an entire cluster to run a model no single chip could hold.
Implementation: Proof of concept
A Kubernetes StatefulSet gives each pod (model shard) a stable identity (model-shard-0, model-shard-1, and so on), which the shards need to address each other deterministically. The key addition, versus data parallelism, is podAffinity, which forces all the pods (shards) onto the same node/node group so synchronization stays on the fastest possible link.
#####################################################
# Basic proof-of-concept; not for production use
#####################################################
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: model-shard
spec:
serviceName: model-shard
replicas: 4 # 4 shards of one model
selector:
matchLabels:
app: model-shard
template:
metadata:
labels:
app: model-shard
spec:
affinity:
podAffinity: # co-locate all shards on a 4+ GPU node
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: model-shard
topologyKey: kubernetes.io/hostname
containers:
- name: shard
image: registry.internal/trainer:1.0
# pods have stable IDs... model-shard-0, model-shard-1, etc
resources:
limits:
nvidia.com/gpu: 1
---
apiVersion: v1
kind: Service
metadata:
name: model-shard
spec:
clusterIP: None
selector:
app: model-shard
Security: Key Considerations
Since model shards sync constantly and latency is a key consideration, any security controls must be implemented with speed in mind. Controls should be hardware-accelerated, asynchronous where possible, and never placed in the critical computation path.
Pod admission. Asynchronous pod admission control should be implemented with both blocking and non-blocking rules. Decide which blocking rules need to be evaluated immediately and which non-blocking rules can be evaluated after pods are created. Cache these policy decisions to avoid repeated evaluation on identical pods.
Communication integrity. Workers exchange data at every compute step, so communication must be tightly controlled and authenticated.
Hardware-level encryption on the interconnect (may come with a bandwidth cost).
Precise network policies limiting communication to only required workers.
mTLS between workers (via SPIFFE/SPIRE), established once at connection time.
Node integrity. Because shards are co-located, a single compromised node can expose several shards of the model at once.
Require secure boot and attestation before a shard is scheduled.
Run workloads on dedicated node pools with taints and tolerations so nothing else lands there.
Add pod-to-host isolation such as gVisor.
Pipeline Parallelism: Scaling the depth
Concept
Models are built from many repeated layers stacked on top of each other. Pipeline parallelism cuts those layers into stages and assigns each stage to a different chip or group of chips. Data flows through the pipeline where the first chip runs the first stage, hands the result to the next, and so on, much like an assembly line. The advantage over tensor parallelism is that chips only communicate between stages, not constantly, so the communication cost is much lower.
The downside is compute utilization. At any moment some stages sit idle, waiting for data to reach them or having already passed their work downstream. Therefore, the whole pipeline runs only as fast as its slowest stage. Balancing the stages well requires real planning about how much work lands in each one, since a slower stage becomes a bottleneck for every chip in the sequence.
Implementation: Proof of concept
A Kubernetes StatefulSet with ordered pod management starts the pods (stages) sequentially, which matters because a pod (stage) cannot initialize until its upstream neighbor is ready. Each stage learns its position from its pod identity and finds its neighbors by their predictable names, and a liveness probe catches a stage that hangs.
#####################################################
# Basic proof-of-concept; not for production use
#####################################################
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: pipeline-stage
spec:
serviceName: pipeline-stage
podManagementPolicy: OrderedReady # start stage 0, then 1, then 2...
replicas: 4 # 4 pipeline stages
selector:
matchLabels:
app: pipeline-stage
template:
metadata:
labels:
app: pipeline-stage
spec:
containers:
- name: stage
image: registry.internal/trainer:1.0
env:
- name: NUM_STAGES
value: "4"
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name # e.g. "pipeline-stage-2"
resources:
limits:
nvidia.com/gpu: 1
livenessProbe: # catch a hung stage and restart
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: pipeline-stage
spec:
clusterIP: None
selector:
app: pipeline-stage
Security: Key Considerations
Data flows from stage to stage, so in pipeline parallelism the handoffs are the main attack surface.
Data integrity between stages. Data is passed between stages and must be protected from tampering.
Enforce strong workload identity (SPIFFE/SPIRE) and mTLS so only authorized stages participate.
Data exfiltration. Each stage handles sensitive data. Strong data residency controls should be implemented to prevent exfiltration.
Minimize data residency per stage. Stages should process and discard data.
Use strict network policies and egress rules so stage pods can communicate only with their upstream and downstream peers.
Denial of service via a stalled stage. An oversized or malformed input can halt the entire line.
Give every stage a timeout, resource limits, and a liveness probe.
Enforce timeouts between stages and at startup so a stuck pipeline alerts and restarts instead of hanging.
3D Parallelism: Putting it all together
Concept
In practice, frontier models don't just use a single parallelism strategy. The industry standard is to use hybrid approaches, which may combine different techniques. A common hybrid is 3D parallelism, which layers all three:
Pipeline parallelism splits the model into stages.
Tensor parallelism splits the math within each stage across multiple chips.
Data parallelism runs the whole pipeline as several copies, each processing a different slice of data.
The cost is enormous operational complexity. You are scheduling work at three levels, each with its own bandwidth requirements, failure modes, and security risks. Every layer you add multiplies the ways things can break, and at these hardware costs, breaking is expensive.
Some systems add a fourth strategy, expert parallelism, on top. We cover it briefly below.
A note on expert parallelism
This post focused on the three strategies that combine into 3D parallelism, but there's a fourth worth knowing by name. Expert parallelism applies to a specific model design called Mixture of Experts (MoE), where the model is split into many specialized sub-networks, and a small learned router sends each input only to the few experts suited to it. MoE models can reach enormous parameter counts because only a fraction of the model activates for any given input, which is why the technique has become common in frontier models.
The experts are spread across chips much like the strategies above, but with a different challenge. The router decides where every input goes, which makes it both the performance bottleneck and the primary security target (crafted inputs can overload a single expert or steer sensitive data toward a compromised one), so the router needs the same identity, rate-limiting, and monitoring discipline you would apply to any critical routing layer.
Conclusion: Wrapping it up
Models and datasets continue to grow, making it important to understand the different scaling strategies and how to secure them. These concepts are not unique to LLM training; many of the same distributed computing and security principles apply to other large-scale systems. I hope this post was able to demystify LLM training and provide you with a solid foundation on how this works in practice; feel free to use the infographic below.
For further and more in-depth reading on the topic, please refer to:
Comments