Best Practices in Learning Measurement
By Jeffrey A. Berk
CLO Magazine, October 2003

Best Practices in Learning Measurement

A top question often asked of organizations creating and revising processes for learning measurement is ‘What are the best practices?’ This question is asked more and more because training departments are being asked to prove their value more and more. In tough economic times all aspects of the organization are being asked to justify their existence, not just training. However, the more prepared training organizations can be to respond to such inquiries the better likelihood they have of maintaining a value added role within the company.

This article attempts to discuss some of the more significant characteristics that have helped organizations create an effective learning measurement process.

Best Practice #1: Plan your metrics before writing survey questions.

First and foremost, never ask a question on a data collection instrument unless it ties to a metric you will do utilze. As simple as this sounds, often is the case where organizations create questions with no purpose in mind.

A great example is a consumer services company that invested enormous resources in custom surveys that were specific to each course with little or no comparability across courses. When asked if the value of collecting this data was beneficial to management, the company stated that while the data was great for course designers, it had gained little interest in the eyes of senior management or other stakeholders.

So, remember never write survey questions or collect data unless it results in a metric your stakeholders find valuable. Once you have finalized your set of metrics, the survey questions are an easy byproduct.

Best Practice #2: Ensure the measurement process is replicable and scaleable.

Organizations tend to spend thousands of dollars on one-off projects to measure a training program in detail. This information is collected over many months with exhaustive use of consultants and internal resources. Although the data is powerful and compelling, management often comes back with a response such as ‘great work, now do the same thing for all the training.’ Unfortunately such one-off measurement projects are rarely replicable on a large-scale basis. So don’t box yourself into that corner.

A classic example is a telecommunications company that hired an expert third party to evaluate a single training program. The third party providing a convincing argument to management that the single program was a good use of company resources. Management quickly mandated the same process be done for all programs. This was not feasible given resource constraints. The training group had boxed themselves into a corner and was forced to quickly come up with a scaled down process that could be replicated and scaled yet they had to backtrack and resell the revised approach to management.

It is essential to make sure that you create a measurement process that can be replicated and scaled across all learning events without spending more on measurement than you do on training. To do this you must acknowledge and accept that not everything needs in depth, precise measures. The key is to use reasonable assumptions to predict and estimate learning effectiveness. Doing so will provide a baseline for you to manage by measurement and extract relevant data points to present to management that clearly demonstrate the value of the learning investments. This is not to rule out having the flexibility in your measurement process to drill deep into a program 5 to 10% of the time where it warrants such an exercise. You simply want to make sure what you do 90 to 95% of the time is replicable and scaleable.

Best Practice #3: Ensure measurements are internally and externally comparable.

Related to best practice #2 is the concept of comparability. It is a significantly less powerful endeavor to do a one-off exercise when you have no base line of comparability. If you spend several months calculating out a 300% ROI on your latest program how do you know if that is good or bad? Surely a 300% ROI is a positive return but what if the average ROI on training programs is 1000%?

A great example is a manufacturer that believed they were an excellent training organization. Day in and day out they measured performance of instructors, courses and facilities. The scores were always consistent. Finally, they then compared against an external group of other organizations and found that they were consistently lower than other training organizations.

Ensuring your measurement process is comparable both internally and externally is critical. Comparing learning effectiveness for each course, comparing investment value by each client grouping, or comparing job impact by key program are just a few examples of how internal and external comparisons can give you a more accurate portrayal of how your training is really measuring up.

Best Practice #4: Use industry-accepted measurement approaches

Management is looking to the training group to lead the way in training measurement. It is the job of the training group to convince management that their approach to measurement is reasonable. This is not unlike a finance department that must convince management of the way it values assets. In both cases, the group must ensure the approach is based on industry accepted principles that have proof of concept externally and merit internally.

An example is a software company that was tasked with the challenge of determining a return on investment for its thousands of learning offerings. Senior management wanted to know the value of the training. The training group researched several methods and approaches looking at books, articles, consultants, and associations such as The American Society for Training and Development. At the end of the day, the company adapted an approach that emphasized the models used by Donald Kirkpatrick in his Four Levels of Learning and Jack J. Phillips in his ROI Process. Senior management was able to more quickly buy into the approach based on the overwhelming external support for these methodologies.

Regardless of the approach you use, keep in mind you need to ‘adapt’ it to your organization, not ‘adopt’ it in your organization. There is a big difference. Adapting it means you take an approach and tweak it to fit your needs. It is not a cookie cutter approach. Finally, make sure you are comfortable with the approach and can defend it. Looking at how others have applied it and its acceptance in the industry will get you more comfortable with the approach and make it more defensible when discussing it with management.

Best Practice #5: Define value in the eyes of your stakeholders

If you ask people what they mean by ‘return on investment’ you are likely to get more than one answer. In fact, odds are you’ll get several. Return on investment is in the eyes of the beholder. To some it could mean a quantitative number and to others it could be a warm and fuzzy feeling.

For example, to showcase the diversity in value let’s look at two very different perspectives. A large utility defines value in measuring the financial, monetary return on every class they run. The value is in comparing the returns to each other and showcasing to management that the benefits exceed the costs of training. Contrast this to a very large oil company. This organization is far less concerned with a financial ROI on training. The company places enormous value on ensuring that the employees receive a quality training experience. The ‘ROI’ for this company is the ‘warm and fuzzy’ feeling they get when they review evaluations and know that the employee was satisfied with the training.

As you can see from the above, had you calculated a financial ROI for the oil company it may have been a waste of your time. However, if all you showed the utility was the ‘warm and fuzzies’ they would have felt your measurement was not adequate. The point is to know your stakeholders. Understand how they define value. Ensure that your measurement process and your resulting metrics yield business intelligence that is of value to each stakeholder.

Best Practice #6: Manage the change associated with measurement

As you can likely see from some of the best practices, they might be doomed for failure if you fail to manage the change with your stakeholders. Successful organizations will spend considerable time and energy planning for the change.

A best practice company is a large, international, manufacturer. The corporate university rolled out a measurement process that would significantly change how the organization evaluates training. To manage this change, the organization slowly built it up during the implementation process. First, the leaders of the corporate university got the buy in from stakeholders by getting them involved in the establishment of metrics and finalizing of data collection instruments. Second, the corporate university sent out communications in its newsletters describing the timetables and benefits of the change. As a result, the organization fully embraced the new measurement process. In fact, the corporate university now receives measurement requests from conference planners and decentralized learning groups within the organization.

In any change it must be managed. Failure to get buy-in from stakeholders can create hostility and resistance to change. Assess the culture and the readiness for change. Plan for change or plan to fail.

Best Practice #7: Ensure the metrics are well-balanced.

Although you want to understand the needs of your stakeholders and have them define how they perceive value, you also need to be proactive in ensuring that your final ‘measurement scorecard’ is well balanced.

One IT organization was told that satisfaction was the sole measure of learning performance. This organization focused its efforts on measuring satisfaction across such items as the instructor, the courseware, and the facility. They received excellent metrics back on satisfaction. However, when it came time to answer the question regarding the impact training made on the job, or its return on investment, the organization did not have the data to support it. Too much emphasis was focused on one element of learning measurement.

A lopsided approach to measurement is risky. Using the popular Don Kirkpatrick model, you can see a balanced scorecard emerging from each of the 4 levels of learning he writes about. The first attribute is reaction. It is important to measure reaction as that can help improve future events and is a proxy for satisfaction. Second is learning. Just because people were happy does not mean they learned anything. It is critical to measure this either through testing or through alternative predictors. Third is behavior. If you learned something can you imply that the person changed their behavior on the job? Probably not that is why you need to measure it. Finally there is results. Did the training positively impact the business results management felt it should have? As you can see, each level is like a quadrant in a scorecard. Some stakeholders may care more about certain quadrants than others but the key is to keep it balanced so that you can comprehensively measure your investment. That is critical in order to continuously improve.

Best Practice #8: Leverage automation and technology

Although this goes hand and hand with a measurement process that is replicable and scaleable it is worthy of separate mention. Your measurement process must leverage technology and automation to do the heavy lifting in areas such as data collection, data storage, data processing and data reporting.

A classic example is a mid-sized accounting firm. The company collected training surveys via paper and then manually keyed them into an excel spreadsheet. Requests for information often took hours to fulfill and could not be done in a timely manner.

In today’s world of automation and technology, any company, large or small can cost effectively leverage technologies such as the Internet to collect data. Even when no computer is in the classroom, surveys can be emailed to participants after the training. With proper reinforcement a decent response rate can be gleaned. In addition, scanning technologies can scan paper data to avoid manual data entry. Finally, software technologies exist to create standardized reports using the collected data. The end result is that you spend less resources collecting, processing, and reporting results and more time analyzing the data for improvement purposes or for showcasing the value of the training back to management.

Best Practice #9: Crawl, walk, run.

When designing a learning measurement strategy it is nice to have a long term vision, but don’t attempt to put your entire vision in place right out of the blocks. The best approach is to start with the low hanging fruit that can be done in a reasonable time frame to prove the concept, demonstrate a ‘win’ and build a jumping off point to advance it to the next level.

A great example is an insurance company that envisioned a measurement strategy that measured across a balanced scorecard of learning, leveraged automation, and provided relevant, timely metrics to key stakeholders. Recognizing the challenge of limited resources and lack of consistent technologies within the organization, this company started out by piloting the process on a few key programs. The pilots afforded the company the opportunity to test drive the process, refine it, and build momentum for expanded usage. Now the company is rolling out the process organization-wide with the goal of integrating various systems involved in the process soon after.

Although hard at times, your first step should not be a step at all but rather a crawl. You can learn a lot from a pilot or test run of your process. Also, you build the quick wins you need to sustain momentum to move the process forward in the organization.

Best Practice #10: Ensure your metrics have flexibility

The last thing you want to do is rollout a measurement process that is inflexible. You will likely have people who want to view the same data but in many different ways. You need to have architected your database to accommodate this important issue thereby creating measurement flexibility.

For example, a large technology training company has multiple stakeholders who want to see the same data from different perspectives because they manage different aspects of the business. The courseware designers need to see the training data sliced by course so they can understand which courses generated the greatest job impact. The team that manages instructors needs to slice the data by instructor so they can monitor the quality levels of their large group of professional instructors. Senior management wants to view the data by location. Each location is like a separate entity and the performance and quality levels of each is of keen importance to them. Finally, the sales and marketing folks want to mine the data for intelligence surrounding historic performance and productivity gains experienced by learners as a result of the training. This helps them create better business cases when positioning the training to prospective buyers.

As can be seen by the above example, you need to carefully think about the data before you collect it. Once collected you are limited with what you got. The saying ‘garbage in is garbage out’ is very true. Most commonly, you should want to ‘tag’ every data element with the following: instructor name (if applicable), learning delivery mode, location where the training was held, learning provider (internal or external), date of training, course name, curricula (group the course belongs to), and program (group curricula belongs to). To make matters easier, technologies such as OLAP cubes can be used to slice this data in near infinite proportions satisfying the needs of all of your data requests.

Finally, flexibility is also inherent in your ability to ‘rollup’ the data. This too must be thought about prior to data collection. Often companies ask different questions for each course. That is good tactical detail but not good strategic intelligence. Different data is more challenging to aggregate up into higher levels. Senior management is far more likely to want to view aggregate data than class or course specific data. So, ensure you have a common set of ‘standard’ questions you ask across all courses that can be aggregated and benchmarked. You can still have course specific questions for the more tactical analysis. In this way you have more flexibility in your data.

Concluding Thoughts:

Edward Hubbard, PhD, consultant and author once said “Training is either at the table working with senior management and adding value or they are on the table perceived as a cost center that is going to get cut.” The best practices mentioned in this article are not an all inclusive list of what it takes to avoid being on the table. However, they should be used as a sobering reminder to us all that not even the biggest and best training organizations are protected against the reality or perception that one’s value can come into question at any given time. Leverage the practices and be a step ahead by creating the right measurement approaches to continuously improve and showcase value to stakeholders.

By Jeffrey Berk, CPA, MBA, Adjunct Professor of Management

Jeffrey is the Director of Products and Services for KnowledgeAdvisors, a business intelligence software company that helps organizations measure and manage their learning investments. Jeffrey can be reached at jberk@knowledgeadvisors.com.