AI/ML gyan:
Data Parallelism
When training very large AI models, a single GPU may not be enough.
There are two common ways to use multiple GPUs:
Data Parallelism → Split the data
Model Parallelis → Split the model
⸻
Data Parallelism
Data Parallelism is a technique where multiple GPUs train the same model, but each GPU processes a different portion of the dataset.
Every GPU has an identical copy of the model.
⸻
Example: House Price Prediction
Suppose you’re training a house price prediction model using 1 million house records.
Instead of sending all 1 million records to one GPU, you divide the dataset into four equal parts.
GPU 1 trains on records 1–250,000
GPU 2 trains on records 250,001–500,000
GPU 3 trains on records 500,001–750,000
GPU 4 trains on records 750,001–1,000,000
Each GPU trains the same model independently on its assigned data.
After one training step, the GPUs share what they learned (their gradients), combine the updates, and synchronize the model weights. Now every GPU has the same updated model, and training continues.
This process repeats until the model is fully trained.
⸻
Real-World Analogy
Imagine a teacher has 1,000 exam papers to grade.
Instead of one teacher grading everything, four teachers each grade 250 papers using the same marking scheme.
After finishing, they combine the results.
Everyone followed the same rules—they simply divided the workload.
⸻
One-Line Summary
Data Parallelism speeds up training by splitting the dataset across multiple GPUs while keeping an identical copy of the model on every GPU.
English
