Processing Large Datasets with Background Jobs
The engineering considerations involved in importing large files without overwhelming the application or the database.
Status: outline. Each section below is a prompt to be answered from real experience — replace with what was actually built, tried, or learned.
The problem
- What data was being imported, how large was it, and who was waiting for it?
- Why could this not simply run inside a web request?
The constraints
- Memory limits, database load, and how long a user was willing to wait.
- What had to be true about progress, resumption, and duplicate processing?
The architecture
- Where does the file come from (object storage), and what picks it up?
- How is a large dataset divided into batches, and who processes each batch?
upload → object storage → enqueue job → worker processes batches
↓
progress · retries · failure log
The engineering decisions
- Chosen: what processing strategy was implemented?
- Considered: synchronous import, streaming, chunked background jobs — what was weighed?
- Why: what made this the right level of complexity?
The challenges
- What happened on partial failure — was a batch retried, skipped, or reported?
- How were memory usage and database pressure kept within bounds?
The outcome
- What size of dataset works reliably now?
- What improved for the person waiting on the import?
The next iteration
- What would change with more time — checkpointing, better reporting, parallel workers?
- What is still manual that should not be?