Multi-tenant ingestion on Lambda from one Lambda per client to one for all

Before: a Lambda per client

Every new client meant setting up a new Lambda that ingested that client's data. Each one had to be maintained, and onboarding took a lot of manual steps.

The argument for this design was isolation: one tenant's heavy data could not slow down another's. That was only partly true. Each file was already ingested by its own Lambda instance, so most of the isolation came from Lambda and not from having separate functions.

It was also inefficient and slow, even at the small volumes it handled at the time.

One codebase for every tenant

The replacement is a single Lambda system. It reads the list of clients and, for each one, where its data comes from. It supports more than one kind of source, SQS and S3.

Two things made it efficient. Records are processed in chunks of 1,000, which took in-memory objects from 80k to 5k and peak memory from 138 MB to 10 MB. Database writes are batched, so inserts go in groups and not one row at a time.

The lower memory use removed the need for a higher Lambda memory tier, which cut infrastructure cost by about 20%.

Onboarding without an engineer

Each tenant's configuration lives in ConfigCat and is created automatically when the client is onboarded. The manual steps for the engineering team went to nearly zero.

That is what let CRIS 1.0 scale to 10 tenants across two models, storm and wildfire.

Notifications, the same way

Notifications had the same problem. Every tenant had its own cron job with that client's settings hardcoded in it.

I moved them to one cron job that runs every hour, reads the notification configuration of every client, and triggers whatever is due. After that there was one piece of infrastructure and one codebase to maintain.

What it did not fix

This was a repair to a badly broken system, and it did its job. But Lambda was never going to be the long-term home for ingestion. We knew that as data volumes grew, it would have to move to a dedicated service with its own CPU.