ACCU Roundup

Just back in from the ACCU conference where I first spoke about my non-blocking FIFO. Most of the crowd was heavily engaged in C++ and I saw a number of great talks ranging from Uncle Bob's keynote on software craftsmanship to some previously unknown speakers that had done some interesting multi-threading library work for C++. And as is the case with most conferences, I managed a hall-way conversation or two.

First hallway of interest was with Uncle Bob. It started from a question he got from the audience about agile about architecture and performance. I'm afraid this is an area that many those claiming to understand agile fall short. Bob put it well when he said (I'm paraphrasing) that planning architecture is like any other planning step, you need to know how much to do and when to do it. Agile does not say skip architecture. From there I repurposed to performance. Again, Agile does not say don't plan for performance, it says, don't tune pre-maturely. How does one avoid tuning pre-maturely? It requires that you match the activities to the stage your project is in.

This description may sound waterfallish but it isn't. It is based on the fact that project pass through different stages that are dominated by different activities. The waterfallish sound to this comes from identifying and listing those dominating activities, requirements gathering, architecture and design, implementation, and deployment. There is a related performance activity or tool activities that fall into one of these categories.

When gathering functional requirements be sure to gather important performance requirements. Performance requirements fall into the category of soft or non-functional requirements. They may not function in mind but they are very important for building an understanding with the user and the development team as to how important response time is for each activity.

The architectural performance tool is a benchmark. The purpose of the benchmark in this context is to ensure that you're architecture support your performance requirements as you understand them. Some products just work better when they are integrated into a project up front. To retro-fit a product that is needed for performance after the fact can costly or you won't receive the full benefit. Agile is great but it's not a magic substitute for certain types of planning. It was great to hear this message from those that know agile.

When you finally get to working with real code the best thing to do is to start collecting performance metrics. This is where Bob and I had a great discussion. It all centered around the idea of granularity. First point, performance testing is *not* unit testing. Where as you can unit test for small amounts of functionality, it is rare that you can performance test small amounts of functionality. Testing small amounts of functionality is akin to micro benchmarking.

Just about any successful micro-benchmark I've been is one that has been written in such a way that it confuses Hotspot so that the code you've written is the code that gets run. Hotspot is tricked to the point where is just doesn't know how to optimize the code. This is not code that you want running in your production system. Thus you simply can't wrap a junit test in a loop and let'er rip. It won't work.

More over, micro-benchmarks tend to focus on CPU to the point where they will saturate them. This is also not a desirable characteristic of performance measurement. One of the things you want to determine in performance testing is the level to which your code will consume hardware resources. One of the questions you want to answer is, will this bit of code play well in that it doesn't hog system resources to the point that other activities suffer. To achieve this it seems sensible that one performance tests at a component level. This performance testing could be piggybacked off of your CI system.

Finally there is the traditional testing before deployment. I don't see this stage as changing much from how it is currently being performed.

Just how effective is this approach. Well it's hard to say because it is very difficult to convince people, especially agile people, that there are benefits to testing for performance requirements. That said, those that have integrated this level of performance testing into their projects have found fewer performance problems when they've gone to production. My argument is, we take testing for functional requirements for granted these days, why should we see benefits from treating soft-requirements with the same level of care. I hope to publish something real on the subject during the summer.

tags: accu  agile  performance