Performance Anti-Pattern: No-stress Testing

The purposed anti-pattern has been given the name “No-stress Testing” as a play on words. The implied meaning is that the test was conducted by a developer that is only interested in a quick check of performance. A check more out of interest than one driven by specific demands or stresses placed on him/her by the organization. After the quick test show satisfactory results, the developer will go back to doing what developers do, develop code. The real meaning of the name implies that the application was not properly load tested using tools that can reliably maintain the same load that would be seen in a production environment. Testing under no-stress conditions is unlikely to reveal any bottlenecks that maybe lurking about.

The refactored solution, stress or load testing, isn't really a solution but more a best practice. As the name suggests, when we test for performance we much ensure that the test places the expected amount of work on the system at a rate that closely mimics that which it will be subject to in a production environment. Applying improper load during testing can hide performance bottlenecks that may only be found once the application has been deployed. 

The refactored solution type is to set load testing requirements and use tools to ensure that these loads are obtained during testing.

The root causes include naiveté, sloth and haste. Setting up and conducting a load or stress test can take a considerable amount of effort. Thus teams under stress to deliver will often start to throw away activities that are viewed as offering lower value.

Traditionally performance testing is tacked onto the end of the projects schedule because it is hard to imagine how one can properly test a partially built system. The naiveté in this thinking is that all performance problems will only manifest themselves in the completed product. While there are classes of performance problems where is this true, there are other classes where a continuous performance testing program would smoke them out.

A common practice with many development tests is to have the developers exercise the system as a group. The problems with the tactic are numerous. While this may have the effect of placing some stress on a system, it is often more artificial than what would be applied with a proper load test harness such as Apache JMeter or (HP) Mercury LoadRunner (to name one open source and one commercial product). In addition, the test is not repeatable. Without a repeatable test it will be difficult to know if the fix actually worked. Further to this point, performance testing is like any other type of testing. We would never consider a functional bug to be fixed until we retested so why do we treat a performance bugs differently.

The anecdotal evidence is “performance was ok when I tested it on our development system" or "when I tested it on my machine". There are many variants to the anecdotal statements but they all point to the same root cause, lack of proper performance testing in a properly configured test environment. 

tags: performance  antipattern  nostress  testing  java