If you can't visit the floor....

One of the things that I like to do when carrying out a performance audit is to visit the people that are using the system. I like to watch how they work and interact with the system. You get to see pain points, jitters and pauses. You get to see natural rhythms, smoothness in workflows. It allows me to remove a layer for abstraction from the numbers being produced by the monitoring tools. I get to evaluate first hand answers to the questions of frequency and duration. What is the most frequent work activity and what work activity takes the longest duration.

Just recently I engaged with a client where it just wasn't possible to visit the users. Fortunately they had a great business analysts and an extremely cooperative management team. More importantly, a development team that was open to inviting criticism from their users. In lieu of a visit to the floor, I was able to send out a quick questionnaire. In an industry that is renowned for bureaucracy I was able to get answer back with-in 24 hours.

First, I'll admit freely to being an amateur when it comes to constructing questionnaires. I know that the questions can affect the answers. I knew this one had to be short, it needed to be non-technical yet provide us with answers to some pretty technical questions. We needed to figure out a way to fetter out performance information from feelings and impressions. I needed to battle against complexity in format and I didn't have much time to get it together. We settled on 7 questions of which I found these ones to be the most useful.

1. Describe any activities that you perform where the system is alway slow to respond
2. Describe any activities that you perform where they system is sometimes slow to respond
3. Are there any times of the day when you find the system is slower
4. Are there any days of the week when you find that the system is slower
5. Are there any activities that you avoid or have trouble completing
6. What action do you take most often when you find things slowing down

The answers to the questions told me that the system was pretty stable and users were overall happy with performance. There were occasions when they experienced problems but they appeared to be random. The solutions in many cases was to logout and then log back into the system.

The interprutation of the answers told me that object churn and sticky sessions were sometimes a problem. Indeed an investigation into these aspects of the system did reveal that though the system was able to cope, these issues could become limits to scalability as they grow their user base.