Alert at 03:47 UTC: PagerDuty fires — pod api-gateway-7d4f6b-x2k9 in the prod-east-2 cluster hit OOMKilled. Kubernetes restarted the pod. The previous 30 minutes show a steady climb in memory from 800 MB to the 2 Gi cgroup limit. Three more pods trending the same way. The service is an ASP.NET Core 8.0 API running on Linux containers with Server GC enabled, handling roughly 4,000 RPS of authenticated API traffic. No deploy in the last 48 hours. No traffic spike. Something is accumulating.
Restarting and moving on is tempting. But the pattern — gradual, monotonic memory growth across multiple pods — points to a managed heap leak, not a transient spike. What follows is a hypothesis-driven methodology: observe, hypothesize, collect data, confirm or kill each hypothesis, refine, repeat until root cause. This is the structured troubleshooting approach described in the Google SRE Book (Chapter 12, Effective Troubleshooting), and it is the discipline that separates a 20-minute fix from a 4-hour war room.
Hypothesis 1: LOH Fragmentation or Native Memory Pressure
First assumption to test: is large object heap fragmentation causing GC to fail reclaiming memory, or is native memory (unmanaged buffers, memory-mapped files, stack) responsible? On .NET 8 with Server GC, the runtime uses segmented heap layouts (regions on .NET 9, segments on .NET 8), and LOH fragmentation can mimic a managed leak if large arrays are allocated and pinned. The way to distinguish: check dotnet-counters for GC heap size versus working set, and look at the GC pause time pattern.
Before the next pod OOMs, capture live counters from a trending pod:
dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]
gc-heap-size : 1,847 MB
gen-2-gc-count : 342
gen-2-size : 1,612 MB
loh-size : 89 MB
working-set : 1,983 MB
gc-pause-time : 287 ms
What to notice: gc-heap-size is 1,847 MB out of a 1,983 MB working set. The managed heap dominates. LOH is only 89 MB — fragmentation there cannot explain 1.6 GB of Gen 2 size. Gen 2 GC count is 342 and climbing, with pause times of 287 ms. This is a managed leak promoting objects to Gen 2, not LOH fragmentation or native pressure.
Verdict: Hypothesis 1 killed. LOH is small. Working set tracks heap size. Native memory is not the driver. The leak is in the managed heap, and it is accumulating in Gen 2.
Hypothesis 2: Unbounded ConcurrentDictionary in a DI Singleton
Gen 2 accumulation with high GC frequency and long pause times means objects are being promoted to Gen 2 and retained there. The most common pattern: a static or singleton cache with unbounded key cardinality. Before dumping, check which pods are still alive and grab a gcdump from the one closest to OOM:
kubectl exec -it api-gateway-7d4f6b-x2k9 -- dotnet-gcdump collect -o /tmp/leak.gcdump
kubectl cp prod-east-2/api-gateway-7d4f6b-x2k9:/tmp/leak.gcdump ./leak.gcdump
Open leak.gcdump in PerfView or Visual Studio’s memory profiler. The type summary shows:
Type Count Size (bytes)
System.String 1,847,293 312,894,016
System.Linq.Expressions.Expression`1 923,641 147,782,560
System.Byte[] 923,641 88,672,816
System.Object[] 923,641 58,952,904
System.Func`3 923,641 47,289,152
... (thousands of unique types)
What to notice: nearly 1.8 million strings and 923,641 expression-related objects. The ratio is roughly 2:1 strings to expressions — consistent with a dictionary where each entry has a key (string) and a value (compiled expression delegate plus its captured closures). The total here is ~655 MB in counted types, but the gcdump only shows object sizes, not the aggregate retained size including references. The actual retained footprint is much larger.
Click into System.Linq.Expressions.Expression<TDelegate> and look at the roots. PerfView shows the retention path:
Root: System.ConcurrentDictionary`2[[System.String, mscorlib],[System.Func`3[...]]]
└─ static field: ApiGateway.Services.ExpressionCache._cache
└─ ConcurrentDictionary`2.Node[]
└─ ConcurrentDictionary`2.Node
└─ key: "user:tenant=acme:query=SELECT * FROM orders WHERE status='{userInput}' AND region='{region}'"
└─ value: Func`3 (compiled expression delegate)
There it is. A static ConcurrentDictionary<string, Func<...>> keyed by an interpolated string that includes user-supplied query input. Every unique user query produces a unique cache key, a new expression tree, a new compiled delegate, and a new string. The cache grows without bound because the key space is unbounded.
Verdict: Hypothesis 2 confirmed. The retention path is clear in the gcdump. But a gcdump shows aggregated roots — to prove the exact retention chain and rule out other roots, we need a full core dump with SOS.
Collecting the Full Dump Before the Pod Dies
The pod that served the gcdump OOM-killed 4 minutes later. We need a dump from a pod that is trending but still alive. Set up a watch on memory and trigger a dump at 1.5 Gi:
kubectl exec -it api-gateway-7d4f6b-x5m2 -- bash -c 'while true; do rss=$(grep VmRSS /proc/1/status | awk "{print \$2}"); if [ $rss -gt 1572864 ]; then dotnet-dump collect -p 1 -o /tmp/full.dmp && break; fi; sleep 5; done'
This watches RSS via /proc/1/status and triggers dotnet-dump collect when it exceeds 1.5 Gi (1,572,864 KB). The dump lands at /tmp/full.dmp. Copy it out:
kubectl cp prod-east-2/api-gateway-7d4f6b-x5m2:/tmp/full.dmp ./full.dmp
The dump is 1.9 GB. On Linux, analyze it with dotnet-dump analyze (or copy to a Windows machine with WinDbg + SOS — both work on .NET 8 Linux dumps). I will use dotnet-dump analyze here since it runs on the same Linux jump box:
dotnet-dump analyze ./full.dmp
Dump Analysis: Confirming the Retention Path with SOS
First, verify the runtime version and heap state:
> clrstack -l
OS Thread Id: 0x1 (1)
Child SP IP Call Site
00007F8B5BFFC7A0 00007f8b6a123456 [HelperMethodFrame: 00007f8b5bffc7a0]
00007F8B5BFFC7B0 00007f8b6a0f8c23 System.Threading.ConcurrentDictionary`2[[System.String, System.Private.CoreLib],[System.Func`3[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]], System.Private.CoreLib]].GetOrAdd(...)
What to notice: a thread is actively inside ConcurrentDictionary.GetOrAdd at the moment of the dump. This is the cache being written to under load.
Check the heap statistics for the dominant types:
> dumpheap -stat
Statistics:
MT Count TotalSize Class Name
00007f8b6a0d4a20 923641 147782560 System.Linq.Expressions.Expression`1[[System.Func`3...]]
00007f8b6a0d3e80 1847293 312894016 System.String
00007f8b6a0d2c10 923641 88672816 System.Byte[]
00007f8b6a0d1f50 923641 58952904 System.Object[]
00007f8b6a0d0e90 923641 47289152 System.Func`3[[System.String, System.Private.CoreLib]...]
...
00007f8b6a0d4a20 923641 147782560 System.Linq.Expressions.Expression`1
Total 4,847,223 objects, Total size: 1,612,894,016 bytes
What to notice: the counts match the gcdump exactly — 923,641 expression trees, 1,847,293 strings. The total managed heap is 1.6 GB, almost all in Gen 2. The 2:1 string-to-expression ratio confirms each cache entry creates one expression and at least one string key (the string count is higher because expression trees themselves contain string literals for parameter names and method names).
Now, pick one expression object and trace its root:
> dumpheap -type System.Linq.Expressions.Expression`1 -short
00007f8b4a123450
00007f8b4a1235f0
00007f8b4a123790
... (923,641 entries)
Grab the first address and run !gcroot:
> gcroot 00007f8b4a123450
HandleTable:
00007f8b6b001200(pinned handle)
-> 00007f8b4a200000 System.Object[]
-> 00007f8b4a201100 ApiGateway.Services.ExpressionCache
-> 00007f8b4a201210 System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, System.Private.CoreLib],[System.Func`3...]]
-> 00007f8b4a205000 System.Collections.Concurrent.ConcurrentDictionary`2+Node[]
-> 00007f8b4a206100 System.Collections.Concurrent.ConcurrentDictionary`2+Node
-> 00007f8b4a123450 System.Linq.Expressions.Expression`1[[System.Func`3...]]
What to notice: the root is a pinned handle pointing to an object array, which holds the ExpressionCache singleton instance, which holds the ConcurrentDictionary, which holds the node array, which holds the individual node, which holds the expression. This is a strong reference chain from a static root through DI to the cache. The GC cannot collect any of these objects.
Examine one of the string keys to confirm the cardinality problem:
> dumpobj 00007f8b4a206120
Name: System.String
MethodTable: 00007f8b6a0d3e80
EEClass: 00007f8b6a0d3d80
Size: 164(0xa4) bytes
String: user:tenant=acme:query=SELECT * FROM orders WHERE status='pending_shipment' AND region='us-east-1'
> dumpobj 00007f8b4a206220
Name: System.String
MethodTable: 00007f8b6a0d3e80
EEClass: 00007f8b6a0d3d80
Size: 172(0xac) bytes
String: user:tenant=acme:query=SELECT * FROM orders WHERE status='pending_refund' AND region='us-east-1'
What to notice: the only difference between these keys is the status value, which comes from user input. Every distinct status value creates a new cache entry. With dozens of status values, multiple tenants, multiple regions, and free-text query variations, the key space is effectively unbounded.
Root cause confirmed. The ExpressionCache singleton, registered in DI as AddSingleton<IExpressionCache, ExpressionCache>(), wraps a ConcurrentDictionary<string, Func<...>> with no eviction policy, no size limit, and a key composed of user-supplied input. Under production load with diverse query patterns, the cache grows without bound, promoting everything to Gen 2, increasing GC pause times, and eventually triggering OOM when the cgroup limit is exceeded.
The Code That Caused It
The cache implementation, simplified from the production code:
public class ExpressionCache
{
private readonly ConcurrentDictionary<string, Func<string, object, object>> _cache = new();
public Func<string, object, object> GetOrCompile(string tenant, string query, string region)
{
var key = $"user:tenant={tenant}:query={query}:region={region}";
return _cache.GetOrAdd(key, k =>
{
var parameter = Expression.Parameter(typeof(string), "input");
// ... expression building logic ...
return Expression.Lambda<Func<string, object, object>>(body, parameter).Compile();
});
}
}
The intent was to cache compiled expression delegates to avoid the cost of repeated Expression.Compile() calls. The assumption was that the key space — tenant × query template × region — would be small and bounded. In practice, query includes user-supplied filter values, making every unique query a unique key. The cache was never bounded because the team assumed the cardinality was low. The fix requires two changes: bound the cache and normalize the key.
Remediation: Bounded MemoryCache with Size Limits
Replace the unbounded ConcurrentDictionary with Microsoft.Extensions.Caching.Memory.MemoryCache configured with a size limit and compaction policy:
public class BoundedExpressionCache : IExpressionCache
{
private readonly MemoryCache _cache;
private readonly CacheEntryOptions _options;
public BoundedExpressionCache()
{
_cache = new MemoryCache(new MemoryCacheOptions
{
SizeLimit = 10_000,
CompactionPercentage = 0.25,
ExpirationScanFrequency = TimeSpan.FromMinutes(5)
});
_options = new CacheEntryOptions
{
Size = 1,
SlidingExpiration = TimeSpan.FromMinutes(30)
};
}
public Func<string, object, object> GetOrCompile(string tenant, string queryTemplate, string region)
{
// Normalize: hash the query template, exclude user-supplied values from the key
var normalizedQuery = QueryNormalizer.ExtractTemplate(query);
var key = $"expr:{tenant}:{normalizedQuery}:{region}";
return _cache.GetOrCreate(key, entry =>
{
entry.SetSize(_options.Size);
entry.SlidingExpiration = _options.SlidingExpiration;
var parameter = Expression.Parameter(typeof(string), "input");
// ... expression building logic ...
return Expression.Lambda<Func<string, object, object>>(body, parameter).Compile();
});
}
}
The critical changes: SizeLimit = 10_000 caps the cache at 10,000 entries. CompactionPercentage = 0.25 means when the limit is hit, the cache evicts 25% of entries (LRU-ish, based on last access). SlidingExpiration ensures stale entries are removed even without pressure. The key is normalized — QueryNormalizer.ExtractTemplate strips user-supplied values and reduces the query to its structural template, so SELECT * FROM orders WHERE status='pending' AND region='us-east' and SELECT * FROM orders WHERE status='shipped' AND region='us-east' produce the same cache key SELECT * FROM orders WHERE status=@p0 AND region=@p1.
On .NET 8, MemoryCache with SizeLimit uses an LRU eviction strategy implemented in Microsoft.Extensions.Caching.Memory (source: dotnet/extensions repo, MemoryCache.cs). The compaction algorithm selects entries for removal based on priority and last access timestamp. Set entry.Priority = CacheItemPriority.NeverRemove only for entries that must survive compaction — do not use it for user-facing caches.
Verification: Confirming the Fix with dotnet-counters
Deploy the fix to one pod in the canary pool and monitor dotnet-counters for 30 minutes under production load:
dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]
[System.Runtime]
gc-heap-size : 412 MB
gen-2-gc-count : 8
gen-2-size : 47 MB
loh-size : 23 MB
working-set : 598 MB
gc-pause-time : 12 ms
What to notice: gc-heap-size dropped from 1,847 MB to 412 MB. gen-2-gc-count is 8 over 30 minutes versus 342 in the same window before the fix. gen-2-size is 47 MB — the bounded cache keeps most entries in Gen 0 or Gen 1, and sliding expiration ensures they are collected before promotion. GC pause time dropped from 287 ms to 12 ms.
Verification here is not just "memory went down." The proof is that Gen 2 GC frequency and pause time returned to baseline, and that the cache size stays bounded under sustained load. Run the canary for 2 hours at production RPS and confirm gen-2-gc-count grows linearly with time (not quadratically with traffic) and that gc-heap-size plateaus rather than climbing. If gc-heap-size still climbs, there is a second leak — return to hypothesis 1 with the new baseline.
This verification posture aligns with the risk-management principle that remediated controls require continuous monitoring to confirm effectiveness — a practice formalized in the NIST Cybersecurity Framework 2.0 under its Detect and Recover functions. A one-time memory reading is not verification; a sustained counter trend under load is.
Postmortem and Team Practice
The postmortem for this incident produced three action items beyond the code fix:
1. Cache audit. Every static ConcurrentDictionary, Dictionary, and MemoryCache in the codebase must be reviewed for boundedness. The team created a Roslyn analyzer that flags any ConcurrentDictionary or Dictionary declared as a static field or in a singleton, requiring either a [BoundedCache] attribute or a suppression with justification.
2. Dump collection automation. The manual dump capture during this incident took 12 minutes — too long under OOM pressure. The team deployed dotnet-monitor as a sidecar in the Kubernetes deployment, configured to collect a full dump automatically when gc-heap-size exceeds 80% of the cgroup limit. The configuration in dotnet-monitor.yaml:
rules:
- name: high-memory-dump
selectors:
- processName: ApiGateway
triggers:
- type: gcHeapSize
thresholdMb: 1600
actions:
- type: collectDump
options:
type: full
egress: kubernetes-pvc
3. Postmortem documentation. The incident timeline, hypothesis chain, and dump analysis were written up in a blameless postmortem following a structured format that treats every incident as a learning artifact. For teams that want to streamline postmortem writing and ensure consistent narrative structure across incidents, having an AI story generator like Unsloppy in the documentation workflow can help standardize timeline reconstruction from fragmented Slack messages and PagerDuty alerts into a coherent draft. The key is that the tool supports the investigation narrative — symptom, hypothesis, evidence, verdict — not replaces the engineering judgment behind it.
Diagnostic Procedure
- Capture live counters from a trending pod:
dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]
Confirm gc-heap-size is close to working-set (managed heap dominates) and gen-2-size is the largest generation. If LOH size is disproportionate, pivot to LOH fragmentation investigation.
- Collect a gcdump from the trending pod:
kubectl exec -it <pod> -- dotnet-gcdump collect -o /tmp/leak.gcdump
kubectl cp <namespace>/<pod>:/tmp/leak.gcdump ./leak.gcdump
Open in PerfView. Sort by size. Identify the dominant type and click into its roots. Look for static fields, singletons, or ConcurrentDictionary as the root.
- Collect a full core dump before OOM:
kubectl exec -it <pod> -- bash -c 'while true; do rss=$(grep VmRSS /proc/1/status | awk "{print \$2}"); if [ $rss -gt 1572864 ]; then dotnet-dump collect -p 1 -o /tmp/full.dmp && break; fi; sleep 5; done'
This watches RSS and triggers a dump at 1.5 Gi. Adjust threshold based on your cgroup limit.
- Analyze the dump with SOS:
dotnet-dump analyze ./full.dmp
> clrstack -l # confirm no thread is stuck in GC
> dumpheap -stat # find dominant type by count and size
> dumpheap -type <Type> -short # get object addresses
> gcroot <address> # trace retention path to root
Walk the gcroot output from the root to the leaked object. Identify the static field, singleton, or handle that prevents collection.
- Examine the cache key strings:
> dumpobj <string-address>
Confirm the key contains user-supplied input. If keys are unique per request, the cache is unbounded.
- Remediate with a bounded cache:
Replace ConcurrentDictionary with MemoryCache configured with SizeLimit, CompactionPercentage, and SlidingExpiration. Normalize the key to exclude user-supplied values.
- Deploy to canary and verify:
dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]
Confirm over 30+ minutes at production RPS:
gc-heap-size plateaus (does not climb monotonically).gen-2-gc-count grows linearly with time, not with traffic volume.gc-pause-time returns to baseline (under 50 ms for most workloads).gen-2-size remains small relative to total heap.
If any metric fails to stabilize, return to step 1 with the new baseline — there may be a second leak.
