While trying to optimize our main backend service running on thousands of cores and TBs of heap, we optimized its run time by 60% by building a massive cache. We're running with such a large heap, and yet ran into multiple obstacles along the way. We had to reinvent what we think of as a cache, and pay close attention to the tiny details that have a major impact when running at such a scale.