Continuous Performance Testing

It's good to finally see Continuous Performance Testing finally getting some press. I figured it was only a matter of time as it is the next obvious step in the evolution of continuous integration and unit testing. Uncle Bob tiwttered a question day. He asked 'regarding estimates,. Do you give estimates that you aren't certain you can meet? Where do you hide the uncertainty".

It's a great question but when does anyone ever give an estimate that they are certain they can meet. Me thinks that this is the common case because rarely do we give serious consideration to the question of how long will it take to ensure this application meets it's performance requirements. In fact, how often do we sit down and really sort out what the performance requirements are.

If this industry has learned anything in recent years it is that holding a big testfest at the end of the development phase of a project is a great way to ensure failure. Why would performance testing be any different? More over, I would argue neglecting to include time for testing is tantamount to professional malpractice.

One of the rallying cries of late has been about avoiding premature optimizations. Unfortunately this has been confused by some as avoiding performance aspects of a project all together only focusing on it when it becomes necessary. In one case I watched a team "agile code" themselves out of a potential optimization that they were going to need to scale. Their argument that they would apply it when needed spoke not of avoidance of applying a premature optimization but more to lack of foresight. This is not what agile is all about.

Each phase of a project is dominated by a particular activity. We still have phases in the project life-cycle that are dominated by requirements gathering, architecture, coding, and deployment. In each of these phases there are corresponding performance related activities. During requirements gathering we need to collect users expectations for performance. We also need to work on simplifying the functional requirements. When the simplify requirements we make them easier to understand and this often leads to simpler code that performs better. When defining architecture we use benchmarking to determine that our decisions will not prevent us from meeting our performance goals. During development we should be looking at integrating CPT on the back of the CI system so that we can gain an understanding of how our system is progressing. I won't speak of the last phase, deployment, because this is something that is already well covered.

In John Smart's awesome book, Java Power Tools, he talks about unit test level performance testing. While John's advice to monitor performance at the unit test level (ie method level) isn't completely wrong, testing at this level of granularity isn't advisable. Performance testing at the unit test level is akin to micro performance benchmarking. At best, it is very tricky to get good results out of an MBM. At worst the results will be useless as the bench will most likely be busted. In fact most MBMs only work after you've added enough code to the test that you've tricked the JIT to not compile out the feature that you want to test. The result is so far from production ready code that.. well.. you get the idea that code to support an MBM is not the code that you'd want to deploy.

Does this mean it is impractical to performance test during development. Absolutely not. What is says is that you need to test at an appropriate level of granularity. Performance testing method.. questionable, far too granular. Performance testing single class is the same as performance testing a single method... Performance testing components... better. Other tidbits; make sure you use a constant sized unit of work and then chart the progression of runtime over time as the component develops. Make sure that you factor our interfering effects due to influences from other components that your component depends upon. If you look after these details, you'll have a better idea of how your application will perform when released into the wild.

Is all of this premature? I had the opportunity to talk to Uncle Bob and Kevlin Henney about how CPT fits into the Agile process. After both discussions it became clear that CPT does have a role in Agile. In fact, including performance activities should only help to decrease the uncertainty in your estimates. After all, I see a lot of cost over runs due to poor performance discovered only after deployment. This costs in a couple of ways. It's more expensive to tune post deployment and it takes resources away from other activities putting those activities at risk. All for lack of performance planning and a lot of it in the name of being agille.

The subject needs much more discussion!

tags: agile  performance  cpt  ci  continuous  integration  java  tuning