Performance Anti-Pattern: No-stress Testing

posted Thursday, 22 November 2007

Proposed Anti-pattern

Name	No-stress Testing
Most Applicable Scale	Application, Component
Refactored Solution Name	Stress testing
Refactored Solution Type	Tooling, Training
Root Causes	Haste, sloth, naiveté
Unbalanced forces	Management of Performance
Anecdotal Evidence	Performance was ok when I tested in development.

Introduction

The purposed anti-pattern has been given the name “No-stress Testing” as a play on words. The implied meaning is that the test was conducted by a developer that is only interested in a quick check of performance. A check more out of interest than one driven by specific demands or stresses placed on him/her by the organization. After the quick test show satisfactory results, the developer will go back to doing what developers do, develop code. The real meaning of the name implies that the application was not properly load tested using tools that can reliably maintain the same load that would be seen in a production environment. Testing under no-stress conditions is unlikely to reveal any bottlenecks that maybe lurking about.

Most Applicable Scale

When considering scale, we must consider the type of performance testing that is most approprate for the phase of the project that we are in. Though the primary scale is integration as it this phase in the projects lifecycle that we can get a complete picture, we can start building a performance picture during development. Thus the most applicable scale is at the level of both application and component.

Refactored Solution Name

The refactored solution, stress or load testing, isn't really a solution but more a best practice. As the name suggests, when we test for performance we much ensure that the test places the expected amount of work on the system at a rate that closely mimics that which it will be subject to in a production environment. Applying improper load during testing can hide performance bottlenecks that may only be found once the application has been deployed.

Refactored Solution Type

The refactored solution type is to set load testing requirements and use tools to ensure that these loads are obtained during testing.

Root Causes

The root causes include naiveté, sloth and haste. Setting up and conducting a load or stress test can take a considerable amount of effort. Thus teams under stress to deliver will often start to throw away activities that are viewed as offering lower value.

Unbalanced Forces

Traditionally performance testing is tacked onto the end of the projects schedule because it is hard to imagine how one can properly test a partially built system. The naiveté in this thinking is that all performance problems will only manifest themselves in the completed product. While there are classes of performance problems where is this true, there are other classes where a continuous performance testing program would smoke them out.

A common practice with many development tests is to have the developers exercise the system as a group. The problems with the tactic are numerous. While this may have the effect of placing some stress on a system, it is often more artificial than what would be applied with a proper load test harness such as Apache JMeter or (HP) Mercury LoadRunner (to name one open source and one commercial product). In addition, the test is not repeatable. Without a repeatable test it will be difficult to know if the fix actually worked. Further to this point, performance testing is like any other type of testing. We would never consider a functional bug to be fixed until we retested so why do we treat a performance bugs differently.

Anecdotal Evidence

The anecdotal evidence is “performance was ok when I tested it on our development system" or "when I tested it on my machine". There are many variants to the anecdotal statements but they all point to the same root cause, lack of proper performance testing in a properly configured test environment.

tags: performance antipattern nostress testing java