Last week I took a class titled "Public Speaking Mastery," a two-day course designed to help people polish their public speaking skills. It was a great course for a consultant who has to speak in front of groups of people all the time and is always looking to hone his skills.

However, the performance measurement part of my brain asked, "How will I know if I actually learn anything from this course? How will I know if I improve over these days?" I wanted to know what the measure was for improved public speaking ability.

However, I knew from the start that it wasn't a typical measure – by typical I mean quantitative. There is no number that tells you at what level or ability your public speaking skills are. I couldn't go into the class and establish a baseline – that I was a 6 – and set a target – that I wanted to be an 8 – when the course concluded.

Set a Baseline, Any Baseline

Of course, the class instructor, having taught similar courses for over 25 years understood this and he was smart about it. He kicked off the two days asking the students what they wanted to get out of the class and people's answers ranged from: "I have this horrible habit of rocking back and forth when I do presentations, I want to stop that" to "I just want to be able to get up and speak in front of a group of people. I cannot do it at all."

So now he had a qualitative data point, or baseline, from each student that at the end of the two days he could go back to and ask about and he did that. At the end of the second day, he found the page on his flip chart and asked each person if they felt they made progress toward their personal objective. It was not an exact science, but it enabled people to determine whether real progress was made.

The other thing he did was set up a video camera in the back of the room. We each had to speak in front of the class three times over the two days and each talk was taped so we could view it ourselves. At the end of the course, we were given a USB drive with our three videos so our progress – or lack of progress – was documented.

Often Times You Will Know Progress When You See It

While I didn't get a score or grade that definitively marked my progress, I was happy with the result. I'm not sure anyone could really develop a qualitative measure that would make people happy tracking progress like this, but I was satisfied with the qualitative measure my three videos provided.

The whole experience reminded me of a lecture from law school on the 1964 Supreme Court case Jacobellis v. Ohio in which Justice Potter Stewart described his test for obscene pornography. Justice Stewart stated that he could never "intelligibly" define hard-core pornography "[b]ut I know it when I see it."

In the same way, there will be occasions during your performance measurement experiences where you will not be able to measure your organization's progress on certain objectives with numbers. Remember, however, that if that objective really is strategically important, a qualitative measure is better than not measuring progress at all because whether you are making progress or not, you will know it when you see it.

• Balanced Scorecard Measurement + Control Charting Theory (November 20, 2010)