Well, we’ve tried working with Varnish and we’ve given up. After desperately trying to make Varnish play nicely with everything else on the system, we’ve given up and removed Varnish from our application stack entirely. Why? Memory architecture.
Part of the documentation on Varnish’s website is a long architectural explanation that the OS should handle what stays in RAM and what gets swapped to disk, and that Varnish thus should not do any memory management as such. There is a problem, here, however. This design means Varnish will basically assume that the OS will handle contention between itself and other programs.
This is not a smart move. First off, some OSes are terrible at that sort of thing. Linux is pretty good. But here’s the real issue; take a database server like PostgreSQL. PostgreSQL correctly lets the OS handle disk caching rather than replicating efforts internally. This is a great move and means that you don’t have to guess how much RAM you can let PostgreSQL take up for disk caching; the OS handles it all. Since it’s just caching, sometimes that space can be reallocated to programs which need some RAM, and later given back to PostgreSQL (or any other app).
varnishd was regularly climbing to around 4-6 gigabytes of RAM usage, forcing even application memory into swap, and completely removing any memory from the OS for disk caching, having a terrible knock-on impact on performance of PostgreSQL on the same machine. I should point out that the 4-6 gigabyte figure was obtained while running varnishd with a 1 gigabyte disk cache.
Basically, if you want to run Varnish (and there are many good reasons to; it’s a fantastic cache server other than this issue) you need a dedicated machine to run it. The architecture of the software makes it impossible for it to coexist on a server with other programs. We even tried having Monit restart it when it reached 1 gigabyte of RAM usage, but it still had a terrible impact and the caching was impacted by it. While having a 45% cache hit on Varnish was a lovely thing, and helped reduce load on our backend servers, it was slowing the backend servers down enough for that to not really work out at all.
With the 1 gigabyte of RAM we freed by removing Varnish, we’ve added four more application servers to EVE Metrics. These are more than coping with demand, and we’re happily seeing things stay nice and stable even with a lot of API accesses. So far, then, so good.
On a side note, users of the popular accVIEW application will be happy to know I’m spending a chunk of time this weekend improving the app and adding some very much needed features, like persistent API key storage for users so that corporate security can be maintained even when people leave corporations or join new ones, forgot password features, and performance improvements.