During Pitru Paksha, our platform runs far above its normal traffic. API requests grow from around 10M to 18M+ a day. Content delivered to users nearly doubles to 19 TB+ a day, and events peak at more than 800K a minute. This load does not pass in a few minutes but lasts for nearly two weeks, with campaign bursts landing on top of an already high baseline.
In this blog, we share how we handle our Tech Infrastructure during Festival Spikes. We start with where the traffic comes from and why a long peak is harder than a short spike. We then look at a time our event pipeline fell behind, and the lesson it taught us. After that, we cover the changes that made the biggest difference, from caching and circuit breakers to image optimisation. We end with the checks we run before every festival.
1. How is our traffic during Pitru Paksha compared to a normal day?
Pitru Paksha brings a significant increase in traffic across the platform. To paint a picture, a typical day compared with a Pitru Paksha day looks like this:
- API requests: Around 10M requests per day normally, increasing to 18M+ per day during Pitru Paksha.
- Content delivered to users: Usually under 10 TB per day across images, videos, audio, GIFs and other assets, increasing to 19 TB+ per day during Pitru Paksha.
- Events: Normally under 55M events per day, increasing to 80M+ events per day, with peaks of more than 800K events per minute.
The event traffic is particularly important from a capacity-planning perspective. While 80M events per day averages out to roughly 930 events per second, a peak of 800K events per minute translates to approximately 13,300 events per second more than 14 times the daily average. This is why averages alone are not enough when planning capacity for a festival period. Our systems need to be designed for peak bursts, not just average traffic.
Where does the traffic come from?
CRM campaigns typically create the sharpest and predictable traffic spikes. A single push notification can bring thousands of users to the home page within a short period. Since these campaigns are planned in advance, we work closely with the CRM team to understand the campaign schedule, target audience and expected traffic. We then scale the required infrastructure ahead of the campaign.
Social media contributes a more consistent stream of traffic throughout the festival, along with occasional spikes that are difficult to predict. For this traffic, the focus is on maintaining sufficient baseline capacity with enough headroom to absorb unexpected increases.
What does this traffic impact?
We broadly think about the platform in three planes. Each plane scales independently, has different failure modes and requires its own health signals.
- Content delivery: Images, videos, audio, GIFs and other assets are delivered through the CDN edge, with an image proxy that optimizes images based on the user’s network and device.
- API serving: APIs that power the application handle user requests, with circuit breakers protecting the system from failures or slowdowns in internal dependencies.
- Event pipeline: Client-side and backend events flow through Kinesis into consumers, Redis queues and downstream workers for processing.
This separation helps us reason about capacity and failures independently. A traffic spike does not necessarily affect every part of the platform in the same way, so monitoring each plane independently is important for understanding where the actual bottleneck is.
2. What makes a two-week peak like Pitru Paksha harder to handle than a short spike?
A short traffic spike is relatively easier to absorb because it lasts only a few minutes. Once traffic drops, caches recover, autoscaling stabilizes, and any temporary backlog can drain.
Pitru Paksha is different. The platform operates at an elevated load for nearly two weeks, with campaign-driven bursts arriving on top of an already-high baseline. Small inefficiencies that may go unnoticed on a normal day can compound over time:
- APIs: Even a few additional milliseconds per request can keep connections and resources occupied for longer, reducing the number of requests the same infrastructure can handle.
- CDN cache: Every request that misses the cache creates additional load on backend services and data stores. A small reduction in cache-hit rate can therefore translate into significant additional load when sustained for hours.
- Event pipeline: If events arrive faster than they are processed, the backlog continuously grows. Unlike a short spike, there is little opportunity for the system to catch up during a sustained peak.
A slow service can be worse than a dead service
Our backend relies heavily on internal service-to-service communication. A single request can fan out to multiple services, which gives us flexibility but also introduces dependencies between services.
Our design goal is simple: if one service becomes unavailable, its failure should not cascade to the services that depend on it.
The chain reaction
A service that fails immediately is often easier to handle than one that becomes slow. A fast failure allows the caller to respond quickly, while a slow dependency can keep resources occupied for much longer.
At Pitru Paksha traffic levels, this can lead to a chain reaction:
- A downstream service starts responding slowly.
- Requests in the calling service remain in-flight while waiting for the response, consuming CPU and memory.
- New requests continue arriving, causing in-flight requests to accumulate.
- Resource utilization increases until the calling service becomes unhealthy or is eventually killed.
- Other services depending on that service are then affected as well.
Without proper protection, a single slow dependency can turn into a much larger platform wide incident. That is why we implemented circuit breakers.
Circuit breakers
We implemented circuit breakers to prevent slow or failing downstream services from consuming resources indefinitely. When a dependency crosses configured failure or latency thresholds, the circuit breaker stops sending requests and fails them fast. This protects the calling service, limits cascading failures, and gives the unhealthy dependency time to recover.
Timeouts are equally important. During Pitru Paksha, we configure timeouts for individual internal calls based on the downstream service’s SLO long enough for a healthy but busy service to respond, but short enough to prevent requests from accumulating indefinitely. We cover the circuit breaker and timeout configuration in more detail in the answer to what made the biggest difference in handling Pitru Paksha load.
The pipeline has to keep up, hour after hour
Behind the application, every tap and screen view generates events from our Android and iOS clients as well as backend services. These events flow through:
Client/Backend → Kinesis → Consumers → Redis → Workers
We use AWS Kinesis as the streaming layer because it is fully managed and allows us to scale the streaming infrastructure without operating it ourselves. We also maintain sufficient shard headroom to handle festival traffic.
However, Kinesis is only one part of the pipeline. The overall throughput is ultimately limited by the slowest stage in the chain.
Two seconds is the line
On the Redis-to-worker side, we closely monitor one key metric:
Events should be picked up by workers within two seconds of being added to Redis.
Queue latency acts as an early-warning signal. When it starts increasing, it indicates that workers are beginning to fall behind, allowing us to scale them before a small delay turns into a significant backlog.
A two-week peak requires sustained throughput and stability not just enough capacity to survive the highest traffic minute.
3. When did the pipeline last struggle to keep up during Pitru Paksha, and what went wrong?
We learned the importance of that two-second line the hard way.
During a previous Pitru Paksha event, processing started to lag. Kinesis was fine with plenty of shard headroom. The consumers were fine too, happily reading, validating and enqueueing events into Redis. The problem was further down: our workers had a fixed maximum concurrency.
As the event rate climbed past what that concurrency could handle, events kept arriving in Redis faster than workers could take them off. The queue grew, and processing fell further and further behind.
| Stage | What we saw |
|---|---|
| Kinesis | Healthy — sufficient shard headroom |
| Consumers | Healthy — reading, validating and enqueueing normally |
| Redis queue | Backlog growing as enqueue rate outpaced dequeue rate |
| Workers | The bottleneck — capped by fixed maximum concurrency |
The lesson was not “buy more of everything”. It was that every stage has to be able to scale on its own, and that the metric to watch is not CPU but incoming events per second versus processed events per second. If the first number is consistently higher, the queue will grow no matter how healthy the servers look.
4. What change made the biggest difference in handling Pitru Paksha load?
There was no single silver bullet. The biggest shift was designing every layer to scale independently while keeping the critical path for user requests and for event ingestion as light as possible. In practice, that came down to six things:
1. Treat latency as capacity
When we expect sustained traffic with sudden bursts on top, our first priority is low-latency APIs. It is tempting to think of latency as a user-experience metric. At festival scale, it is a capacity metric. The faster each request completes, the more concurrent users the same infrastructure can serve — and the less likely a burst is to snowball into timeouts and retries.
2. Start with the hot path
A handful of APIs carry a disproportionate share of traffic: the ones called every time a user opens the app or lands on the home page. When a CRM campaign fires, these endpoints take the hit first. Before every festival, we identify them and optimise them, specifically less work per request, fewer downstream calls.
3. Cache aggressively, wherever needed
Our rule at festival scale is simple: cache aggressively wherever it is needed. Anything that reads far more often than it changes like home page content and other hot-path responses is a caching candidate. Every request served from cache is one that never touches a downstream service or a database.
We apply this at every layer:
- Redis for API responses and frequently reads data on hot-path endpoints.
- CDN edge for images, videos, audio, GIFs and other assets.
- Optimised image variants from our image proxy, cached at the edge so each one is generated only once.
Why 95% is the number
For APIs, we hold the Redis cache hit rate above 95% at all times. It is easy to underestimate what a few percentage points mean, so here is the same traffic at different hit rates:
| Cache hit rate | Requests reaching the database | Database load |
|---|---|---|
| 99% | 1 in 100 | 0.2× |
| 95% (our floor) | 1 in 20 | 1× |
| 90% | 1 in 10 | 2× |
| 80% | 1 in 5 | 4× |
Dropping from 95% to 90% sounds minor but it doubles the load on everything behind the cache. Therefore, we pay close attention to first-class production metric like the Redis cache hit rate which we believe differentiates a fast app from a degraded one
4. Fail fast: circuit breakers and SLO-based timeouts
To stop the chain reaction, every critical internal call goes through a circuit breaker. It watches calls to a downstream service and, when that service becomes unhealthy, it stops sending traffic so the caller fails fast instead of waiting.
- Closed — normal operation; failures and timeouts are tracked.
- Open — the failure threshold is crossed; calls are rejected immediately, protecting the caller’s CPU and memory.
- Half-open — after a cool-down, a few trial calls go through. If they succeed, the circuit closes; if not, it opens again.
The failure stays contained to the unhealthy service, its dependents stay responsive, and the struggling service gets room to recover instead of being hammered with more traffic.
Timeouts set by SLO, not by habit
A circuit breaker is only as good as the timeout behind it. During Pitru Paksha, we set the timeout for each internal call according to the SLO of the downstream service. It is one of the most important settings we review before the festival.
- Too long, and requests pile up holding CPU and memory — the exact failure above.
- Too short, and healthy-but-busy services get treated as failures, tripping circuits for no reason.
- SLO-aligned, and each call waits exactly as long as the downstream service is expected to take, and no longer.
At scale, waiting is not free.
5. Keep content at the edge, and fit it to the network
Ours is a content-heavy app. Images, videos, audio and GIFs make up most of what a user downloads, so content delivery is our largest source of outbound data. During Pitru Paksha, we serve 19 TB+ of content a day to users including images, videos, audio, GIFs and more. This is up from under 10 TB on a normal day. That growth is expected but at that volume, the average transfer rate alone is about 1.8 Gbps, with much sharper peaks during campaigns.
The same principle applies here: cache aggressively and keep traffic at the edge. Every CDN miss is an origin fetch and extra latency for the user, so edge hit ratio matters just as much as Redis hit rate does for APIs. We also track data transfer out as both a capacity signal and a cost signal, since it nearly doubles during the festival.
Images that respect the user’s network
Our users connect over everything from fast broadband and 4G/5G to slow, patchy connections. Sending the same image to everyone means either a sluggish app on slow networks or wasted bytes on fast ones.
So our images go through an image proxy server that optimises each image based on the user’s internet tier. Slower connections get lighter, more compressed images that load quickly; faster connections get higher quality. During Pitru Paksha, this pays off three ways:
- Faster loads for everyone, especially on constrained networks where content-heavy screens feel slowest.
- Fewer bytes sent, which helps keep the 19 TB+ of content we serve each day in check.
- Cacheable output — optimised variants are cached at the edge, so the proxy does the work once per variant, not once per request.
Because the proxy sits on the cache-miss path, its capacity matters most when new festival content goes live or a campaign sends users to images that are not yet cached.
6. Keep ingestion boring, and scale every stage on its own
Our consumers deliberately do very little. They read events from Kinesis, validate them, and enqueue them into Redis. That is it. Heavy processing stays out of the ingestion path, so ingestion stays fast even under load.
Consumers process events in batches, which cuts per-event overhead and lifts throughput. When we expect a higher event rate, we can raise the batch size but bigger batches cost more CPU and memory, so batch size is always tuned alongside consumer computes. We also make sure consumer capacity keeps pace with the number of Kinesis shard.
Because Kinesis, consumers and workers each scale on their own, we can add capacity exactly where the queue-latency signal says it is needed, which is precisely what the worker incident taught us.
5. What should a team check first before the next Pitru Paksha?
Before every major festival, we walk through the whole platform against expected, peak and sustained traffic. It starts with the traffic itself, expected API requests and peak requests per second on hot-path APIs, events per day and per minute, CDN transfer, CRM campaign schedules and audience sizes, and then moves layer by layer:
| Layer | What we check | The question we must answer |
|---|---|---|
| APIs | Hot-path endpoints optimised; p95/p99 latency under load; autoscaling limits; downstream calls per request | Can hot-path APIs stay fast at peak burst? |
| Service calls | Circuit breakers on critical calls; timeouts set per downstream SLO; thresholds and cool-downs | Can any one service fail without hurting its dependents? |
| Redis cache | Every hot-path API reviewed for caching; hit rate; memory headroom; TTLs and warm-up | Will the hit rate stay above 95%? |
| CDN & image proxy | Edge hit ratio; cache headers on festival assets; proxy capacity on misses; bandwidth and cost | Can the edge serve 19 TB+ of content a day? |
| Kinesis & consumers | Shard headroom; consumer-to-shard capacity; batch size; consumer lag | Can consumers read as fast as Kinesis receives? |
| Redis queue & workers | Queue depth and latency; worker concurrency; incoming vs processed events/sec | Is queue latency staying under 2 seconds? |
During the festival itself, our dashboards have to answer a short list of questions at a glance: Are hot-path latencies and error rates on target? Is the cache hit rate holding? Are any circuit breakers open? Is the CDN serving from the edge? Is Kinesis lag or Redis queue latency rising? And after each burst, is every layer recovering? Festival traffic is not something we can eliminate or perfectly predict. The goal is not a system that never sees a spike. It is a system where spikes are expected, absorbed and forgotten by the next morning.
Resilience at festival scale is not about having the biggest infrastructure. It is about knowing where the bottleneck will appear, seeing it early, and being able to scale exactly that part before anyone notices.
