Seeing through hardware counters: a journey to threefold performance increase | Netflix TechBlog

You don't often get to see performance debugging down at the level of the JVM, but this is a pretty interesting read. The problem is framed up:

Although we confirmed fairly equal traffic distribution between nodes, CPU and latency metrics surprisingly demonstrated a very different, bimodal distribution pattern. There is a “lower band” of nodes exhibiting much lower CPU and latency with hardly any variation; and there is an “upper band” of nodes with significantly higher CPU/latency and wide variation. We noticed only ~12% of the nodes fall into the lower band, a figure that was suspiciously consistent over time. In both bands, performance characteristics remain consistent for the entire uptime of the JVM on the node, i.e. nodes never jumped the bands. This was our starting point for troubleshooting.

Ultimately comes down to how the cache is laid out in physical memory!