How to Analyze Learning Data: Metrics for a Balanced Approach
ASTD (Online) OD/Leadership News, June 2006

By John Skinner, KnowledgeAdvisors

There are numerous metrics that serve specific purposes. Some of the most critical include performance metrics, operations metrics, financial metrics, and cultural metrics. As the names indicate, performance metrics deal with the actual performance of learning programs relative to a set of criteria, operations metrics describe what is going on in the organization, financial metrics catalogue the investments made, and cultural metrics tell the story about the overall organization.

How to Collect This Data

Once you understand these four categories of metrics, you have several possible methods to collect data, including surveys, interviews, focus groups, and empirical research. Depending on your particular situation, each method has its advantages and disadvantages.

Surveys are easy to scale. Items on the survey should conform to the metrics of interest—and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.

Focus groups are valuable sources of information as they allow you to dig deeper with follow-up questions. The most important thing to remember when using focus groups is structure; you should maintain a list of questions that conform with your metrics and follow it. The drawback to focus groups is that they are time intensive and are not as easy to scale.

Empirical research can serve as a goldmine of information. While this method can be time intensive, the results often are highly credible. Some examples of empirical research include control-group studies or statistically linking training to quantifiable organizational outcomes. You may need to engage an outside consultant to help with an empirical study, but this can be worth it for costly, visible, and/or strategic programs where credibility and precision in the resulting metrics are paramount.

Interpreting the Data

Once you collect the data, you must properly interpret it, aggregate it, and populate the desired metrics.

Aggregation refers to the level you choose to examine the data. For example, if you want to look at a level 1 metric, instructor performance, you can look at the data at the class level: What was the performance for this particular class? You also can look at this from the instructor level: What is this instructor's performance overall across classes? Having a clear idea of what levels of aggregation may be needed for various metrics before data collection begins is important.

Aggregation can be thought of as filtering the data. You can only filter, or aggregate, based on certain criteria that you collect with the rest of the data. Knowing which metrics need to be aggregated and presented at what levels ahead of time can save you headaches down the road.

Frame of Reference

Even with the best metrics and data, they should not be interpreted in a vacuum. It is essential to establish context for quantitative metrics.

Benchmarking against an external data set from similar learning programs can put metrics in perspective. This information can be crucial to maintaining a human capital edge in a competitive industry. Internal benchmarks also can provide context for metrics.

Finally, the combination of internal and external benchmarks can help set goals. Goals themselves can be an important context for evaluating the performance and effectiveness of the learning organization, as well as a process check when initiating improvement plans.

As you can see, it is crucial to have a strategy in place to effectively leverage learning metrics. If you use the information presented here as a guide to setting your strategy, you will be much better positioned to adapt and scale to meet the needs of your organization and its stakeholders.

John Skinner is the manager of analysis services for KnowledgeAdvisors, a learning analytics technology company. You can reach him via email at jskinner@knowledgeadvisors.com.