“This is our bulk-download feature” said my Team Lead to me. I went back through the log expecting to find one bad decision I could point at and say "there, that's what took so long." Instead, I found a chain of dominoes consistantly falling, one after another.
Fix the storage problem, and execution time becomes the problem. Fix that, and now multiple workers are stepping on each other's state. Fix that, and the completion logic starts asking questions one counter can't answer.
So this ended up being less about rewriting one feature and more about the way I was thinking through the design as I went
The first version was already sitting there when I picked this up, an intern's first pass at the problem:
one Lambda, download everything to /tmp zip it, upload it.
And it worked.. at least, it worked for the requests it had been tested against.🤓
My job was to take that and make it actually hold up at real scale. Everything from here on is what happened when it didn't.
If i restate it more clearly
The requirements sounded simple enough:
Up to 50 files per request.
No limit on individual file size or total download size.
Downloads are org-scoped.
Each org has a monthly quota.
That second requirement was the one that mattered most.
“No cap on size” sounds harmless when you're writing a requirements document. we’ll come to this later. The interesting part wasn't the final architecture. It was how the architecture changed every time the problem changed.
I was being asked to support downloads with no size limit, while our compute environment had hard limits on disk and execution time.
One requirement that shaped everything
There was one part of the requirement that sounded simple but actually dictated a lot of the design.
The user shouldn't have to download 50 files separately.
A bulk download could contain dozens of files, sometimes many gigabytes in total. The experience we wanted was:
select files → click download → get an archive, not 50 files
So the output of the system wasn't "move these bytes somewhere." It had to produce something the user could actually download.
One thing I want to be upfront about, because it changes how the rest of this reads: a large request doesn't come back as a single archive. It comes back as a small number of parts part-1.zip, part-2.zip one per chunk, each a complete and independently valid ZIP.
That was the trade. Producing one archive from N parallel workers means every worker has to write into predetermined byte ranges of a single object, and the archive's layout has to be computable before any bytes are written. Which is exactly what the two articles below make possible and exactly the complexity I decided this feature didn't need.
That sounds obvious, but once I started thinking about making the system faster, I ended up going pretty deep into how a ZIP actually works. I came across two articles from the AWS Community Builders that were particularly useful:
S3 zipper challenge: a parallel zip assembly that beats the single Lambda approach
and its follow-up:
Zipping 15GB of S3 files in 6s. How the power of community made it possible.
The first article got me thinking about a detail of ZIP that I hadn't really cared about before: STORE mode. STORE mode doesn't compress the files. It stores the bytes as-is, and because the layout is deterministic, you can calculate where each file will sit in the final archive ahead of time. That makes the archive much easier to assemble in parallel.
The second article took that idea further with UploadPartCopy: instead of downloading a large S3 object through the worker and uploading those bytes again, S3 can copy the object data directly into a multipart upload.
I wasn't going to copy either architecture. But reading them changed the way I was looking at the problem.
The ZIP itself was a requirement. How quickly we could produce that ZIP was the engineering problem.
V1 to V2: Streaming makes it faster. It doesn't make /tmp bigger.
When I first started looking at the bulk-download feature, my goal was pretty simple:
make it faster.
The first version was downloading the files and then building the ZIP. I started thinking about streaming instead. If we could stream the files as we downloaded them rather than waiting around for entire files to finish, we could start processing them sooner and avoid doing unnecessary buffering.
That seemed like the obvious next step.
Then, while thinking through the approach, I came across a request that was failing for a completely different reason.
One client had more than 10 GB of data in their drive. crazyy
The reason was pretty straightforward: Lambda's /tmp storage has a 10 GB limit, and our implementation was trying to put the entire ZIP there before uploading it.
So First, let's make this scalable. Then we'll make it faster. the classic advice is also
“make it run, make it right, make it fast”
Streaming is useful for performance. It lets us process data as it arrives instead of waiting for the entire thing to be available first. But if the destination is still /tmp, streaming doesn't solve the actual problem we had.
So V2 wasn't about making it faster , it was about removing /tmp from the equation entirely.
V2 to V3: nothing touches disk
Files get read from S3, piped through archiver, and the output goes straight into an S3 multipart upload, part by part, with nothing ever written to /tmp. No intermediate file, no intermediate buffer that grows with the request size.
Peak memory for a 2.86 GB file: 283 MB. And that number holds up under different loads, not just that one file: across 4 separate runs, from a 365 MB request up to 3.14 GB, from a single file up to 47 files, throughput stayed in the same 19–21 MB/s band every time. That consistency is itself the finding. If per-file overhead mattered, more files at the same total size would move the number. It didn't. Whether it was one 2.86 GB file or a folder of dozens of small ones, the pipeline moved bytes at the same rate the bottleneck was the single stream itself, not the archive step, not the request shape.
When a request fails partway through, the multipart upload needs to be cancelled. But the order matters. If archive.abort() runs before the multipart upload is aborted, the archiver ends its stream cleanly. lib-storage sees that clean stream termination as a successful end of input and goes ahead and completes the multipart upload.
No error.
Just a truncated ZIP sitting in S3 that looks completely valid from the outside.
The fix was simple but unintuitive: abort the multipart upload first, then let the archive stream close. Otherwise, the failure path quietly turns into a successful upload of a broken archive.
V3's next limit showed up immediately after we removed the disk constraint.
A single Lambda invocation can run for at most 180 secs (what we had configured) the max AWS allows is 900 sec on a good day. So while the streaming pipeline could now handle arbitrarily larger data from a storage perspective, a large enough request could still take longer than the Lambda's maximum runtime and would fail.
We had removed one hard limit and exposed the next one.
By this point I'd already gone down the ZIP rabbit hole once. I knew there were much more aggressive ways of parallelizing ZIP assembly, including STORE mode and S3-side
UploadPartCopy. But for this feature, I wasn't trying to build the world's fastest ZIP implementation. I needed to get one large job out of a single Lambda invocation.
The problem wasn't that the streaming implementation was too slow. The problem was that we were still asking one Lambda to do all the work.
V3 to V4: fan out, and now you have two problems
v4 was the answer to the lambda’s clock : stop doing the work in one Lambda. Split the job into chunks, dispatch each chunk to its own Lambda over SQS, let them run in parallel. One orchestrator works out how many chunks a request needs and hands them off. N workers zip and upload their piece independently.
But once I started thinking about that design, the problem got a little more interesting.
now a good lesson how not to fall in the pit of over engineering while designing this distributed fan out approach i was also thinking about how would we know that a particular req submitted by user which now has been divided into N workers each worker would need to know how many parts are still pending and which number is this it may be very much possible that that the 44th part completed before the 37th one and marked the job completed these were all the thoughts i was going through..
This introduced a bunch of questions I hadn't had to think about before.
Those were the right questions to ask. But there was another trap here: making the chunking itself more complicated than it needed to be.
Working out which files should go into which chunk took a few tries.
If the next file is 3GB, it has to stay together. I can't put 1GB in one worker and the remaining 2GB in another just to make the numbers look nice.
So I started making the chunker smarter, added a hard upper bound of 4GB, special handling for oversized files, a lookahead at how much data was still left in the request, and eventually a multi-bin best-fit approach where each file could go into whichever open chunk it fit into best.
I also tried sorting the files largest first.
The thinking was that putting the big files in first would give the smaller files more chances to fill the gaps, which should mean fewer chunks.
It did the opposite.
For the same amount of data, the descending-order approach ended up needing more Lambdas, which meant more cost, without giving me a useful improvement in completion time.
So at that point I was making the chunker increasingly sophisticated to solve a problem I didn't actually have.
I didn't need perfectly balanced chunks.
I needed chunks that were independent enough for multiple Lambdas to work on them in parallel.
So I simplified it.
Keep one chunk open. Process the files in their original order. Keep adding files. Once the chunk's running total has reached the target, start a new one.
for file in files:
if current is null or current.bytes >= CHUNK_TARGET:
current = new chunk
current.files.push(file)
current.bytes += file.sizeThe important detail is that the check is on current.bytes, the running total already sitting in the chunk.
It's not checking whether the file about to be added would cross the target.
That makes the target exactly what it sounds like: a target, not a hard ceiling.
If adding a file takes the chunk from 1.3GB to 2.2GB, that's fine. The chunk closes after that.
And if the next file is already 3GB, it gets added anyway. The chunk becomes oversized, and the next file starts a fresh chunk.
The rule doesn't need to know anything about the incoming file's size. It only cares about how much is already sitting in the current chunk. And that was enough.
On a real request with 50 files and 13.8 GB total, the work split into three chunks. The three workers ran for 150.93s, 147.59s, and 47.83s. Wall clock for the whole job was 152 seconds, compared with roughly 463 seconds when the same work ran through one worker serially.
So the chunks weren't perfectly balanced. They didn't need to be.
The important part was that the work was no longer stuck inside one Lambda. That part worked.
Every worker, when it finished, now had to report back: which chunk it was, what it processed, how much it processed, and whether it succeeded. That result had to land somewhere.
And that's where things got messy. That tradeoff was intentional. I was okay with some uneven chunks in exchange for a chunker that was easy to reason about.
V4 to V5: when parallel workers start stepping on each other
Fan-out solved the execution-time problem, but it introduced a different one.
Now one job could have N workers finishing independently, and all of those workers had to report their results somewhere.
My first version put every worker's result into one shared array on the job row.
When workers finish one after another, that looks completely fine:
Worker 1 → [key1]
Worker 2 → [key1, key2]
Worker 3 → [key1, key2, key3]The problem starts when two workers finish at roughly the same time.
Worker 2 reads [key1].
Worker 3 also reads [key1] before Worker 2's write has landed.
Now both are working from the same old value.
Worker 2 adds key2 and writes:
[key1, key2]Worker 3 adds key3 and writes:
[key1, key3]Whichever write lands last wins.
So now key2 is gone from the job's state, even though Worker 2 actually finished successfully and its output is sitting in S3.
That's a nasty kind of bug because nothing crashed. We just silently lost part of the state.
That was the problem we had to fix.
My first idea was to stop treating the results as one array.
The orchestrator already knew how many chunks it had created, so it could assign each worker its own key when dispatching the chunk. Worker 2 gets key2, Worker 3 gets key3, and instead of rewriting the whole array, each worker adds its key to a DynamoDB set.
So the operation changes from:
read → modify → write to: atomically add my key to the set
Two workers adding different keys don't overwrite each other.
My Team Lead was on board with that. My Engineering Lead wasn't.
The concern was the dynamically assigned keys. The orchestrator decides those chunk identifiers at runtime, which means the shape of what gets written isn't fixed ahead of time. That makes validation and schema guarantees harder to reason about.
So we changed the shape again. Instead of going the cost-effective way we choose to follow good practice..
Instead of storing every chunk's state inside one job row, we created a second table where each chunk gets its own row, keyed by job_id and chunk_index.
I also checked what DynamoDB could actually handle here, because now I potentially had a lot of workers finishing in parallel. A single physical partition can handle about 1,000 WCU/sec of writes, so concentrated writes are something you don't want to ignore.
The new shape helped here too. Every worker was writing its own
(job_id, chunk_index)item instead of everyone hammering the same shared item.I also put a practical limit of 1,000 chunks per request. With a roughly 2GB target per chunk, that gives us an approximate upper bound of 2TB per request.
~2GB × 1,000 chunks ≈ 2TB.
Now Worker 2 and Worker 3 don't touch the same piece of state at all.
Worker 2 writes:
(job-123, chunk-2)Worker 3 writes:
(job-123, chunk-3)There's nothing to read first. Nothing to merge. Nothing for one worker to overwrite.
And this fixed another problem I hadn't hit yet.
SQS gives us at-least-once delivery, so a completion can be processed more than once. With the new model, a duplicate completion for chunk 2 simply tries to insert the same:
(job-123, chunk-2)The conditional insert says the row must not already exist. It does. So the duplicate gets rejected.
That meant the new table wasn't just fixing the concurrent-write race. It also made the completion record idempotent.
One structural change closed two different failure modes:
Two workers overwriting each other's state
The same completion being processed twice
And it's worth being precise about why the second table is the better model, because it isn't really about tables.
The job row answers questions about the whole request. Each chunk row answers questions about exactly one piece of work. Once those were separated, workers stopped competing over the same mutable item not because I added a conditional write, but because there was nothing shared left to compete over.
Job table Chunk table
PK ORG#<org_id> PK job_id
SK BULK_DOWNLOAD#<job_id> SK chunk_index
status status SUCCESS | FAILURE
total_chunks output_key
completed_chunks bytes
resolved_chunks file_count
transferred_at started_at
TTL 35 days finished_at
error
TTL 7 days
My Engineering Lead's objection to dynamic keys is what pushed me here.
But the reason this is the better design is the state ownership, not the schema validation.
There's one more case the conditional insert covers, and it took me a while to see it. A worker can succeed while its completion message is delayed long enough that the DLQ handler has already decided the chunk failed permanently. Both paths write to the same (job_id, chunk_index), and both write conditionally. So whichever terminal outcome lands first wins, and the late one is rejected. Every chunk resolves exactly once as a success or as a failure, never both.
One counter can't answer two questions
Splitting the chunk state into its own table fixed the race. It didn't fix the question of when a job was actually done.
The first version had two numbers:
completed_chunks
total_chunksThe completion Lambda was connected to the Job Table's DynamoDB Stream through a Lambda event source mapping. So whenever the job row changed, DynamoDB Streams produced a record and the Lambda was invoked. The stream is set to NEW_AND_OLD_IMAGES, so every record carries the item's state both before and after the write, not just a diff the Lambda gets a full snapshot of what the row was and what it just became, in the same invocation.
The logic was simple. Whenever it ran, it checked:
completed_chunks == total_chunksIf they matched, we knew every chunk had succeeded. The completion Lambda could then read all the chunk rows, copy the results into the final result, delete the chunk rows, and mark the job COMPLETED.
That worked for the happy path.
Then a chunk failed.
Say there are five chunks and chunk 2 fails permanently. Now the state looks like:
completed_chunks = 4
total_chunks = 5The failed chunk isn't going to increment completed_chunks. So that condition can never become true.
The other four chunks will still finish. The DynamoDB Stream will still keep producing records. The completion Lambda will still get invoked. But every time it checks:
4 == 5And nothing happens.
We needed another signal. Not "did every chunk succeed?" We already had that. We needed: "has every chunk reached a final state, whether it succeeded or failed?" That's where resolved_chunks came in.
Now there were two counters:
completed_chunks → incremented only when a chunk succeeds
resolved_chunks → incremented when a chunk succeeds
or permanently failsSo the completion Lambda could now react to either condition.
If:
completed_chunks == total_chunksthen every chunk succeeded. Read the chunk rows, copy the results, clean them up, and mark the job COMPLETED.
But if a chunk fails, completed_chunks will never reach total_chunks. Instead, as every chunk reaches a terminal state, resolved_chunks keeps moving. Eventually:
resolved_chunks = 5
total_chunks = 5Now we know that every chunk has reported an outcome. The job is already marked FAILED when the first permanent failure happens, so this signal isn't there to decide the job's status. It's there to tell the completion flow: okay, every chunk is done. Finish the cleanup and audit work.
So the two counters were answering two completely different questions:
completed_chunks → Did everything succeed?
resolved_chunks → Has everything finished?One counter couldn't answer both. Two counters could.
What it actually is, start to finish
A client selects files and hits download. The backend checks quota and whether that user already has a job running, writes a row to the job table with status PENDING, and hands back a job id.
That insert triggers a stream event. An EventBridge Pipe filtering for INSERT drops a message onto an SQS queue. An orchestrator Lambda picks it up, claims the job, resolves every file, sizes them, and packs them into chunks the running-total rule from earlier, no fixed ceiling. It seeds total_chunks and both counters, then puts one message per chunk on a second queue.
Each chunk gets its own worker Lambda: stream every file from S3 through archiver, straight into a multipart upload, nothing touching disk. On success, it writes its own row to the chunk table conditioned on that row not already existing and in the same transaction, bumps both counters by one. On permanent failure, after exhausting retries and landing in the dead-letter queue, a separate handler writes a FAILURE row instead, bumps only resolved_chunks, and marks the job FAILED right there.
Every write to the job row triggers the completion checker off that same stream. If completed_chunks has reached the total, every chunk succeeded it reads every row back from the chunk table, writes the results, marks the job COMPLETED, deletes the rows. If resolved_chunks has reached the total but completed_chunks hasn't, every chunk is accounted for and at least one failed already marked FAILED the moment it happened, so this just writes the audit record and cleans up.
The frontend polls the job row every 10 seconds until it lands on one of those two states, then shows the download.