Failures in optimization and how we learned to let go

One thing, done well - which means optimization

We started this app with one mission, which we already wrote about in why we open sourced Nona: build software that does one thing and does it well. Everything in Nona is done. The whole app is production ready. We do not want to add features, we do not want feature bloat, we do not want a thousand small toggles, and we do not want to staple AI onto things that never asked for it just because we had an idea that might work.

So if you are not adding features, what do you do? You strengthen the foundation. You make the thing less shitified and more focused on doing its job. For us that meant one obvious first step: resource usage.

We got a baseline and faced the facts

Before touching a line of code we built a benchmark harness and ran the full matrix - both storage providers, 1 / 100 / 10,000 keys, 1 and 50 concurrent users. Here is the SQLite half of it.

Everything normal is great. One key, fifty users: 8,397 req/s at 80MB. A hundred keys, fifty users: 6,469 req/s at 105MB. Then the bottom row - 10,000 keys with 50 concurrent users - and we are at 1,175MB average, 1,331MB peak, with a p99 of 7.5 seconds. That was the row we decided to go to war with.

Attempt 1: is it SQLite's fault?

First suspicion: the database layer. Our repositories talk through a small abstraction so that SQLite and SQLD can be swapped, and that abstraction is shaped like the libSQL wire format. Which means every row that comes out of a local SQLite file gets dressed up like a JSON row first.

var rows = new List<LibsqlRow>();
while (await reader.ReadAsync(ct))
{
    // one dictionary per row, 7 entries each
    var values = new Dictionary<string, object?>(columns.Length, StringComparer.OrdinalIgnoreCase);
    for (var index = 0; index < columns.Length; index++)
    {
        // boxes every value, and turns TEXT into a UTF-16 string
        values[columns[index]] = reader.GetValue(index);
    }

    rows.Add(new LibsqlRow(columns, values));
}

For 10,000 rows that is 10,000 dictionaries, 70,000 boxed values, and - the expensive part - every value turned into a .NET string. Our values are stored as UTF-8 on disk and go out as UTF-8 on the wire, but .NET strings are UTF-16. So 16KB of data on disk costs 32KB in memory, purely for the round trip.

So we went the other way entirely: drop Microsoft.Data.Sqlite, talk to SQLitePCLRaw directly, and never create a string at all. Read the column as a span of bytes straight out of SQLite's own page buffer and write those bytes into the JSON writer.

while (raw.sqlite3_step(stmt) == raw.SQLITE_ROW)
{
    // ReadOnlySpan<byte> pointing into SQLite's page cache - no allocation
    writer.WritePropertyName(raw.sqlite3_column_blob(stmt, 0));
    writer.WriteStartObject();
    writer.WritePropertyName("value"u8);
    writer.WriteStringValue(raw.sqlite3_column_blob(stmt, 1));
    writer.WritePropertyName("contentType"u8);
    writer.WriteStringValue(raw.sqlite3_column_blob(stmt, 2));
    writer.WriteEndObject();
}

And it worked. On our heaviest fixture the isolated data layer went from 3,188MB allocated to 1,621MB - a 49% cut - with peak working set down 53% and CPU down 47%. Wired into the real endpoint it held up: average CPU across the 50-user run dropped from 22.5% to 10.5%, and peak memory came down about 25%. Byte-for-byte identical responses, verified at both dataset sizes.

The catch is what we gave up. Connection pooling, statement lifetime, error mapping, the memory management that Microsoft.Data.Sqlite quietly does for you - all of that becomes ours to own, by hand, forever, and only for the SQLite provider. We banked a genuine win and a genuine maintenance bill in the same commit.

Attempt 2: stream SQLite straight into the response

This one felt obvious. If the problem is holding a big response in memory, then do not hold it - read a row, write a row, flush, forget it. We pushed IAsyncEnumerable all the way from the SQLite reader through the repository and the query handler into the endpoint.

writer.WriteStartObject();

await foreach (var pair in values.WithCancellation(cancellationToken))
{
    writer.WritePropertyName(pair.Key);
    writer.WriteStartObject();
    writer.WriteString("value", pair.Value.Value);
    writer.WriteString("contentType", pair.Value.ContentType);
    writer.WriteEndObject();

    if (writer.BytesPending >= FlushThresholdBytes)
    {
        await writer.FlushAsync(cancellationToken);
        await bodyWriter.FlushAsync(cancellationToken);
    }
}

writer.WriteEndObject();

We proved the pipeline genuinely streams. First row arrives at 15% of total query time, and the managed heap sits at 1.3MB halfway through a 10,000 row scan. The data layer was doing exactly what we asked.

And the process memory went up by 25%.

Because the API buffers regardless of what we do. The bytes leave our code and land in Kestrel's pinned buffer pool, which sizes itself to peak concurrent demand and then keeps it. Streaming made each request take longer, which meant more requests overlapped, which grew the pool further. We had optimized the producer and handed the problem to the socket layer. Fifty concurrent users pulling a large environment will spike, no matter how elegant the code upstream is.

So we looked at where the memory was hiding

At this point we stopped guessing and took the process apart with GC dumps and counters, right after a heavy run. This is the part that changed our minds.

gen0 / gen1 / gen2          0.8 / 6.5 / 78.2 MiB    <- our actual data
LOH                         174.8 MiB
POH (pinned)              5,048.0 MiB               <- Kestrel socket buffers
--------------------------------------------------
Live GC heap              5,308.3 MiB
GC committed but EMPTY    6,255.1 MiB               <- GC hoarding address space
Non-GC native             6,618.5 MiB               <- SQLite, sockets, runtime
--------------------------------------------------
Working set              18,182.0 MiB

Look at the first line. Our data - the actual config we were asked to return - is 85MB. Everything else is machinery. Five gigabytes of pinned socket buffers. Six gigabytes of heap the garbage collector committed and never gave back, because the machine had memory available and Server GC will happily take what it is offered. Six gigabytes of native allocations outside the GC entirely.

The enormous number was mostly the runtime helping itself to an unconstrained box. When we ran the exact same Native AOT image in Docker with a 1GB hard limit, the same workload for a realistic config did 1,246 req/s in 159MB with zero errors, and it idled at 40MB. We changed nothing in the code. We just gave it a memory limit.

We checked it from every angle, including the tempting ones

We were not going to trust one environment. We ran it framework-dependent and as Native AOT. On Windows and on Linux. Bare process and containerised. We benchmarked empty ASP.NET projects to find the floor. We tried different GC flush thresholds, different buffer sizes, server and workstation GC.

And yes, of course, we had the conversation. Should we rewrite it in Go? In Rust? Oh boy. Somewhere out there a man is already halfway through typing “this is why you should have used Rust” without having read a single number in this article. He has typed it under a CSS post. He has typed it under a Postgres post. He has never met a problem that was not, deep down, a borrow checker deficiency. We love him. He is wrong.

Look - we understand the wishful thinking. No GC, no committed-heap hoarding, problem solved, blazingly fast, memory safe, rewrite it in a weekend. But go through the decomposition again with us. Rust removes the 6GB of empty committed heap - genuinely, that one is real, well done. It does not remove the five gigabytes of pinned socket buffers, because every language ever written has to hold bytes on their way to a socket. It does not remove the native SQLite and kernel buffers, because that is SQLite and the kernel. Optimistically you turn 18GB into 8-10GB, which is still nowhere near a 256MB target, and you got there in nine months instead of an afternoon.

Meanwhile the arithmetic that breaks you is language independent: a large response multiplied by fifty concurrent clients is a large number of bytes that have to exist somewhere at the same time. The borrow checker has no opinion on this. Rust has to hold them too. A rewrite would have cost us months and bought us a smaller number that is still the wrong number.

And then we asked what we were optimizing

Let us reiterate the goal here. We want Nona to run on a 256MB server, as small as we can reasonably make it. Arguably we could push for 128MB, but let us not exaggerate in the age of modern computers.

So here is where we had to stop and be honest with ourselves. What is the scenario we have been defending for weeks? It is a user with 10,000 keys, who fetches all of them, constantly, from 50 clients at once, forever, with caching switched off. Reckless is a polite word for it.

That is not a user. That is a denial of service attack against your own server. And we had quietly accepted a requirement that we must absorb that attack, not crash, and cheerfully return all ten thousand keys every time. This is madness.

It is also - and we say this with affection for our past selves - optimization for the sake of optimization. Not because it is a real use case. The benchmark that produced our scariest numbers explicitly disables conditional requests. In reality the endpoint already returns an ETag, and we measured 2,000 consecutive polls from a real client: 2,000 responses of 304 Not Modified, at 248 polls per second, with zero memory growth. The steady state was never the problem. We built a monster fixture and then went to war with the monster.

What we kept, and what we let go

We cut our losses. We reverted the streaming work entirely - it was slower and hungrier and more complex, which is an impressive hat-trick. We parked the zero-copy SQLite rewrite too, and that one hurt, because 47% CPU is a real number that we measured more than once. But it buys a second hand-maintained data path to protect against a scenario only a madman would create. And once you are down that road there is always another turn to take - we could fork SQLite and patch it ourselves, we could rewrite the whole thing in Rust and claw back another third of the memory because there would be no GC at all. What we gained by stopping is our own sanity, which is also a resource worth optimizing.

And then we landed somewhere that felt genuinely freeing: screw it. Go break your own server. Nona is self-hosted. It runs on your box, on your metal, under your memory limit, with your configuration in it. If you decide to put ten thousand fat keys in one environment and then hammer it from fifty clients with caching turned off, you will have a bad afternoon, and that afternoon belongs to you. We are not going to contort the architecture to save you from it.

What we owe you instead is honesty about the numbers. Give the container a limit and the runtime stops helping itself. Here is the whole thing, measured on the Native AOT image in a Linux container capped at 1GB - every row in its own fresh container so nothing bleeds into the next one.

It idles at 61MiB no matter what you put in it. A hundred keys hammered by fifty clients peaks at 123MiB while doing 1,617 req/s. So 256MB is a comfortable home for a normal configuration, with room to spare - and that is the same binary that helped itself to 18GB when we gave it a 64GB box and no instructions.

The last row is the cliff, and we are leaving it visible on purpose. Ten thousand fat keys pulled by fifty clients will not fit in 1GB, and the runtime starts returning 500s rather than dying. Now you know where the edge is. Go stand near it if you like.

What we learned

We were justified in trying, and we would do it again. That is the job. You take a piece of software and you tinker with it until it works properly and behaves the way you expect it to, and you do not get to skip that part because it might turn out to be unnecessary. Every engineer reading this has burned a week on something that went in the bin. That is how you find out.

But as some famous scientist in some comic book probably said, we were so busy working out whether we could that we forgot to ask whether we should. We just started optimizing, and we kept going for weeks, because there was always one more thing to try and one more graph that looked slightly better than the last one.

We got something out of it anyway. We know this codebase far better than we did a month ago, we have a benchmark harness we trust, and we can point at a number now and tell you exactly which layer it came from. Every attempt we made was backed by a perfectly reasonable story about where the memory was going, and every one of those stories was wrong until we finally took the heap apart. Streaming should have helped. It did not. Rust should have helped. It would not have.

Next up we are doing this properly for latency. Unlike a memory ceiling that nobody is ever going to hit, latency is something every user feels on every single request. That one we are not letting go of.