The results of a recent online survey Ascendant conducted to learn about the strategy and performance management practices of mission-driven organizations started me thinking about the subjectivity and effectiveness of surveying people as a way for gathering measure data.

As proponents of the Balanced Scorecard method of strategy and performance management, Ascendant helps clients develop measures for tracking their organizations' progress. A large part of this is starting the measurement process as soon as practicable, so we often urge clients to use whatever data they have as soon as they can get it rather than waiting until they can get perfect data for perfect measures. In addition, organizations' that are just ramping up this process may have to rely on subjective survey data to have anything at all to measure in certain areas.

The answers to the following question from our survey drove my thinking: How would you rate your organization's overall effectiveness versus your peers (those in the same social sector within your community)? Fifty-seven percent of respondents said their organization is performing above the level of their peer organizations, while only 8% admitted to being below peer group performance.

Since it would be a statistical anomaly (to say the least) for 57% to be "above average," how reliable are survey results such as these? Is such a result due to "self-selection," where mostly high performers respond? What are the implications for mission-driven organizations trying to get their measurement process up and running?

Survey Respondent Self-Selection?

For example, a school district that wants to measure how well students believe they have been prepared for college and career will have to survey former students to gather data for the measures. However, the issue the school district runs into when relying on such data is the same you and I encounter when shopping online and reading customer reviews of products we are interested in purchasing: There has to be a certain amount of self-selection of the respondents, doesn't there?

Most people who write reviews of a product either really hate it or really love it, and often more of the former than the latter, right? Wouldn't the most useful reviews be from those in the middle who aren't responding emotionally? So, doesn't the school district have the same problem when relying on student surveys?

Drill into Responses to See What Drives Them

Yes, the school district does have the same problem. However, it also has a similar solution. While it may be easy for an online consumer to simply rely on the average number of stars the Summer Infant Video Monitor gets from Amazon customers (4.5 stars out of 5.0), we get a lot more useful information from reading the reviews as well. Finding out what, specifically, customers liked and disliked about the product enables us to make a more informed decision about purchasing it.

In the same way, the school district needs to ask students more than whether they felt prepared for college and career. It needs to ask what the school district did to prepare them and what it failed to do.

Just as an online consumer may learn that a camcorder's simplicity and lack of "extraneous" features was a plus for some people while it may have been a negative for others, the school district may learn that programs some students liked, others did not. And, more importantly: Why?

Some students may have liked the idea of the program, but thought it was executed poorly and recommend fixes. Therefore, cutting it may not be the right solution. Acquiring this type of information will enable the organization to focus on what it is doing right and what needs to be fixed. Then it will be able to develop the right initiatives to start improving performance.

So, if you are using surveys to gather measure data, remember to include the open-ended, follow-up questions that may provide the real insight into how your organization is performing. This information may prove to be invaluable in improving your performance and helping you achieve your strategic goals and objectives.

• Balanced Scorecard Measurement + Control Charting Theory (November 20, 2010)