Performance tuning is about applying localized optimizations

I always start my talks with the following disclaimer; "The resemblance of any opinion, recommendation or comment made during this presentation to performance tuning advice is merely coincidental". The slide never fails to both amuse yet cause people to wonder why they decided on my session. Why do I start with this disclaimer? The simple answer is; performance tuning is about local optimizations and before you apply any advice given by any manual, guide or self proclaimed expert, you need to poke your head up, take a look about to see if it will work for you in your current situation.

Case in point. I was working with an application that had some severe performance problems. After solving the easy one, I wanted to see if I could decrease the disruptive effects of GC and improve application responsiveness and throughput by using the concurrent collector. Everything that I'd read to that point suggested that I needed make this change. It had worked in every other situation where I'd applied it. However in this instance, it caused response times to suffer making the situation even worse than before. That brings me to my second point. What is the criteria for deciding on the success or failure of a particular optimization.

My guide to understanding when an optimization has worked is when you actually feel a positive difference in the performance of the application. As simple and obvious as this sounds, it is incredible how quickly it is forgotten in the heat of the battle. If you are not making life better for your users, you need to roll back on the "optimization". With customer X that is exactly what happened, I rolled back from the less disruptive CMS to a more disruptive one. The net effect was a better user experience. A more useful question is, why was the more disruptive collector better for performance.

At an abstract level, a computer system is nothing more than a series of queues. Response times in this system are guided by Little's Law. The applicable part of Little's Law states that throughput is inversely proportional to service time. Placing more load on a system will have the effect of increasing service time due to the effects of queuing which will in turn increase latency. Putting less load on a system should have the opposite effect. But in this case overall response time was a sum of the response times of all the little subsystems (queues). If you decrease the latency in one subsystem you will increase the pressure on downstream subsystems (queues). The net effect on response time will be determined by the difference between the decrease due to the performance improvement and the increase due to the increase in downstream pressure. If the downstream component's performance is close to the tipping point, the overall performance degradation could swamp out the positive benefits gained from tuning the upstream part of a system.

In less abstract terms, CMS isn't a good choice for a collector if your system relies on another system (external or otherwise) that is on the tipping point of it's performance curve. The reason is, the less disruptive CMS collector allows application threads to reach any downstream bottleneck faster than it normally would. Its what I'd call a hurry up and wait condition. Once the downstream bottleneck pushed past the tipping point, it will add more to the response time budget than any gains made by the performance improvement. What I did was throttle application's threads with a less efficient garbage collector in such a way that users experienced an overall improvement in response times.

The side bar to this is if your application is in this condition, it needs help, serious help. Also, you are in a highly localized condition where the normal tuning advice is unlikely to help and in fact may hurt. These types of performance problems are often very confusing to work through. This is especially true when you've done the right things and response times still suck or have gotten worse.

One of the things that hope people bring away from my performance tuning course is how to evaluate performance advice to know if it could be applied in the situations they will be facing. I'm happy to say that my May offering is now full. I'm looking forward talking about performance and then hitting Kalathos, a favorite haunt of mine. For all of you that have missed on this offer, I'll be repeating it at the end of September.

tags: java  performance  training  cms  gc  response  times