Learning Measurement: It's not How Much You Trained, It’s How Well You Trained
By Jeffrey A. Berk
The eLearning Developers Journal, November 3, 2003
A few weeks ago I was talking to a colleague, Toni Hodges, who is an independent consultant that helps organizations with learning measurement. She and I were discussing the difference between what I referred to as ‘activity measures’ vs. ‘performance measures’ and she quickly summarized my terms into ‘how well you train vs. how much you train’ and the terminology stuck with me. So, this article explores this very issue: focus training measurement on how well instead of how much. Quality and impact versus quantity and butts in seats is what matters in today’s world. Astute users of metrics will be the first to understand the difference so those who produce the metrics should be making adjustments in their measurement plans to ensure the right metrics are being gathered.
Activity measures vs. Performance measures
It seems sensible to start with the difference between activity measures and performance measures. Activity measures tell you how much you have trained. Examples of activity measures include average number of students per class or facility utilization. A more comprehensive list of activity measures is included in table one below.
Performance measures tell you how well you have trained. Examples of performance measures include time to job impact, change in strategic results isolated to the training, or instructor performance. A comprehensive list of performance measures is included in table two below.
Activity measures tend to be easier to collect than performance measures. Most learning management systems can gather such data although it is debatable how well they perform at reporting the data for useful analysis. Activity measures are primarily data components of the training registration process. Because training needs to be coordinated and schedules the data is there to data mine. Most organizations do a fairly decent job of reporting some activity measures that are easy to obtain such as number of classes ran, or number of people trained.
Performance measures are more challenging to obtain. Some feel that certain performance measures are nearly impossible to obtain. Satisfaction-type measures such as instructor performance or courseware quality can be obtained through end of class evaluations often referred to as ‘smile sheets.’ Knowledge transfer can be obtained through pre and posttest scores but the testing exercise can be a significant drain on a company’s resources often not making it a practical tool to use across the board for all training. Still yet, training impact on the job and on specific business results baffles many with not only how to do it but how to do it without spending more on the measurement exercise than the cost of the training itself. Finally, ROI, often thought as the holy grail of all training measures seems so distant to many training groups it is nothing more than a dream.
Nonetheless, performance measures are critical. If you don’t know which programs had the greatest impact on the job and the company’s business objectives your measurement system has some significant chinks in its armor.
Call to Action: Performance Metrics Needed
The new millennium is a completely different workplace than even a short time ago. In the late nineties the economy was booming, the money was flowing, and training had a blank checkbook to experiment with all kinds of new programs. It was not all bad, eLearning spawned as a result of this and continues to grow year after year as a valuable learning delivery method.
However, the bubble burst. The dot com hey day is over. The economy dried up and senior management went with a back to basics way of doing business. At the same time massive layoffs occurred and departments were asked to do more with less. Finally, all budgets have been under a microscope of scrutiny. The training department is not immune to this treatment. As such, the training department can no longer continue to justify its existence with solely activity measures.
Think of this, the CEO of a company is trying to control spending and costs. She asks the chief learning officer for a training budget. That budget is prepared and put forth. The CEO wants some evidence to justify not cutting the budget this year. Reasonable business decisions dictate that tools that help the business perform better and improve the bottom line are good investments. The chief learning officer tells the CEO that they trained 80% of the workforce in the prior year, that nearly 300 classes were taught and that anecdotal evidence suggested people really liked the training. As the CEO, this is not a reason to renew a significant budget line item.
Why was the CEO not compelled by the metrics the chief learning officer gathered? They are quantitative and you can make pretty graphs out of them and do all kinds of statistics on them. The reason is because the CEO was not told that the training supported the business objectives of the company. Did the training help increase revenues? Did it help decrease costs? What kind of impact did it have on the average employees job performance relative to the salary paid to the employee? None of that data was collected, measured and presented.
The bottom line is the chief learning officer’s budget is on the table to be cut. If the right performance metrics had been gathered all along, the chief learning officer would be sitting at the table with the CEO talking about the next training program that is predicted to increase customer satisfaction or reduce cycle time.
Senior executives care about the tools that help make a difference on the job and on the business results. Articulating that your training department has just reached a new all time high for the number of times Typing 101 was taught is not a compelling metric to a senior business leader looking to see how each budget line item adds value to their business.
Hopefully the example above sticks. Hopefully it creates a compelling story to gather performance metrics. Although more challenging, it can be done. But how? Remember you’re living in a world where your budget to conduct your core business, training is at risk. You’re not about to allocate significant resources that are already at risk to an expensive measurement solution that may only measure 5% of your programs. What do you do?
Choose Offense Rather than Defense:
Corporate universities may have found themselves in the situation I just described above. The worst case you can be in is when you are on the defensive. This is the scenario where you got your budget cut because you did not showcase the value of training as a strategic tool to improving job performance. The marketing group retained their budget. They did analysis showing how advertising improved sales. The purchasing group showed how their vendor management tactics saved costs. But the training group failed to show how they improved the human capital they trained.
So now the search is on. Determined not to let this happen, to not sit idle, a best practice-training department sees what is happening around it. They know other functions within the organization are proving value. Those other departments are figuring out how to practically showcase their value through scaleable, replicable, and practical measurement solutions. They are not only doing this to justify their existence and mitigate budgets at risk, but to use such measures as tools to continuously improve and proactively showcase value to senior leaders.
The best practice chief learning officer wants to be on the offensive. This person knows it is not prudent to keep providing the same activity metrics from years past that don’t work in today’s business world. They also know that if senior management forces the issue they’ll be on the defensive, which is not a good position to be in.
So, the best practice chief learning officer will search for the solutions. Key ingredients of the solution include taxonomy, process, methodology and technology. All of these ingredients are absolutely critical to put in place a cost effective yet compelling learning measurement solution. But, it is either that or sleepless nights worrying about budget risks. What would you do?
The Search is On: Finding the Right Model
I have had the fortunate or unfortunate experience of seeing managers in the learning department devise measurement strategies. I have authored a few myself as well. The first step in the process is to research learning measurement taxonomies and models. After all, why start with the blank sheet of paper when clearly others have solved this problem before you were faced with it. In doing your research you are likely to run into many models by PhD’s and other really smart people who claim their approach is the right one. However, one model that has stood the test of time is the Learning Levels model by Dr. Donald Kirkpatrick. It started as a series of articles created in the 1950’s, books were written on them and today, organizations such as the American Society of Training and Development and Training Magazine’s Top 100 use the models as critical pieces to their objectives.
Readers of this article will probably be able to cite Kirkpatrick’s 4 Levels of Learning Evaluation, lets review them briefly just for fun.
Level One: Reaction. This answers the question, ‘Did they like it?’ If done right, it can be a set of performance measures for all the pieces that comprise the satisfaction component of a learning program.
Level Two: Learning. This answers the question, ‘Did they learn?’ Since training is in the business of transferring knowledge and skills to their customers it is only fitting to measure performance in this area. Just because someone had a satisfying experience does not mean they learned new skills or knowledge.
Level Three: Behavior. This answers the question ‘Did they use it?’ A learner could have learned significant new knowledge but not been given the opportunity to apply the training on the job, hence it is wasted training as such this is a key performance metric.
Level Four: Results. This answers the question ‘Did it impact the bottom line?’ Probably the most important measure to a CEO or anyone outside of training. It really looks at the business results changed by training.
The Kirkpatrick model is nice, but without a process to really measure these levels it might not be practical. Enter Dr. Jack Phillips. Phillips is another individual you’ll run across when doing research for your measurement solution. He has had two major contributions to the field of learning measurement that will help you in your learning measurement solution. First, he built a process to measure the Four Levels. Second, he added a fifth level, ROI. Combined, his ROI Process is used around the word as a tool to help learning organizations measure Kirkpatrick’s Four Levels, and Phillips’s fifth. You can see a brief overview of the Phillips’s ROI Process in Exhibit One.
As a way to ensure your measurement process is accepted by your team and by management, leveraging over 80 years of industry accepted models such as Kirkpatrick’s Learning Levels and Phillips ROI Process is a smart move. It is significantly easier to get buy in from stakeholders by suggesting a solution that is industry accepted as opposed to something you drew up on the back of a napkin during your lunch hour.
But, you’re not out of the woods yet, not by a long shot. If you read the books by Drs. Kirkpatrick and Phillips or attend certifications on the ROI Process endorsed by the ASTD, you’ll quickly conclude that it is a lot of work. After all, nothing that is worth doing is easy. The ROI Process has been accepted and works because it is rigorous, conservative and solid. The Learning Levels have been embraced by so many because of their unique balance of performance metrics ranging from satisfaction to linkage to business results. So putting what is right into practice is the next step but if it takes a significant investment of financial, physical and human resources, that is not something you can easily sell to management. Asking for more money to measure training than the cost of training itself would not be prudent. So what can be done?
Enter technology. The new millennium, if nothing else, has taught us how to leverage technology so we can do more with less. The next section will discuss how to use technology as an enabler to practically implementing industry accepted measurement models.
Technology Wrapped Around Methodology:
Any measurement process aims at collecting data, storing it, processing it and reporting it. That is what needs to happen to measure Kirkpatrick’s Four Levels, Phillips’ Fifth Level and to work your way through the Phillips ROI Process.
With this in mind, why not leverage technology to streamline the steps so you don’t have to see significant outlays of resources for your measurement solution? Let’s start with data collection. There are a host of inexpensive web-based data collection tools. These tools are easy to use and allow you to collect data from learners, instructors, managers, and any stakeholder at any point in time. Leveraging the Internet to collect data saves costs of paper processing. It is worth investigating and leveraging where practical and feasible. Even if paper is necessary, scanning technologies can easily scan quantitative and qualitative data into databases for centralized data storage and processing.
When it comes to data storage, I don’t mean that metal file cabinet in the basement closet of your office building that so many end of class evaluations go to as their final resting home. Simple tools like spreadsheet applications can be inexpensive and powerful technologies where data collected online can be dumped for efficient data processing. More sophisticated solutions include using local relational databases such as Access that is more powerful than Excel or using enterprise relational databases such as SQL or Oracle that can span the organization easier. Data storage in technology tools is central, secure, and makes processing easier than that basement file cabinet.
Data processing is probably the most time intensive piece in all of measurement. The key here is to have flexible tools that allow for the building of queries that can then be automated or standardized. OLAP tools such as Cognos or Microsoft Analysis Services are very powerful for querying large amounts of data. However, a strong word of caution. Do not throw technology into the hands of the functional users untrained on learning measurement. Allowing a line of businessperson to write their own OLAP query that compares learning data to business data can be counter productive if not dangerous if they do not know what they are doing.
A have to pause in our story to provide an example here. I read a recent article of a sales department reporting an ROI on training at over 10,000%! At first glance that is phenomenal. A deeper discussion revealed that there was no attempt to isolate a revenue increase to training, it just so happened that revenues increased at the same time sales training occurred. The data showed this correlation and hence the training department attributed the sales rise to the training. What about the economy, the competition, technology, people, process and a host of other factors that could have accounted for the sales increase? This was a classic case of uneducated users being handed a data processing tool to write their own queries. Not something I would suggest doing.
Back to the article. If you have the right people writing appropriate queries that a novice user can then run reports against, you can leverage OLAP tools appropriately. In doing so, you can help a diverse set of stakeholders by creating queries that result in reports targeted at the right level of person for the analysis.
This leads to reporting. In most organizations there is a need for 4 primary types of reports. Your measurement solution needs to be able to provide reports for each. These reports include tactical, aggregate, executive, and value analysis.
Let’s start with tactical reports. These are for the staff personnel in the training department that need to manage the learning investments on a day-to-day basis. Examples of tactical reports include class evaluation summaries, respondent qualitative verbatim, and the actual evaluations themselves. These are used to spot problems before they can get any bigger.
Next are the aggregate reports. Middle managers in a learning group use these reports. They aggregate tactical data for monitoring and quality control. For example the person responsible for managing the sales curricula needs to quickly view all the course titles and determine which are most effective. The person managing the eLearning content needs to easily see how effective it is compared to traditional instructor-led training. These managers need to see tactical data rolled up and they need to be able to filter it down too. Reports for these folks should do this.
Then executives in the learning group need reports. They need to see comparisons internally and externally. For example how did employees in each line of business that fund the training see the training impact their jobs? How did the key indicators of training compare to external benchmarks? And what exceptions occur on a day-to-day basis? Executives need summary, exception-based data that is comparable internally and externally.
Finally there is value analysis. This is packaging it all together. It is your balanced scorecard that you can present to management when the budget is up for renewal or you need to fund a new program. It’s the measures you have an appointment with CEO to review on a quarterly basis cut by eLearning vs. ILT so the CEO can see how that eLearning investment is doing. Value analysis sums it all up. It presents all the measures in an easy-to-generate and easy-to-interpret scorecard. Value analysis allows a chief learning officer to sit down with a C level person or a line of business manager to cover how training impacted the job, how human capital was improved through training, and yes, the financial return on training in hard and soft dollars. Metrics such as benefit to cost ratio, payback period and ROI percentage are in the value analysis.
Examples of some of the reports mentioned above can be found in Exhibit 2 and 3. Exhibit 2 shows how actual data can be compared to performance goals and external benchmarks for each level of learning for continuous monitoring of the Learning Levels. Exhibit 3 shows the balanced scorecard that can be presented to management to showcase value in learning investments.
A Solution to Work 100% of the Time:
We’ve talked about how to take solid models by Kirkpatrick and Phillips and wrap technology around them for your measurement solution. But, you need to ensure that your solution has flexibility. Most of the time you can get by with reasonable indicators to manage your business. These don’t have too be in depth and dead on accurate but provide enough confidence to allow people to make intelligent decisions. Designing the right data collection instruments targeted at the right stakeholders at the right time is important to building a model that is easy to use but provides reasonable outputs. Recognize that you might want to go deeper when the program is strategic or cost a lot. Thus your solution should allow for more in depth measurement to occur.
Because of this need for flexibility, we have devised a three dimensional model that can help provide measurement solutions that work with 100% coverage. These models are the learner-based, manager-based, and analyst based solutions. You get good breadth in measuring all 5 Levels of learning measurement in all of these models. But, the depth increases as you move from learner-based to base to analyst-based. The cost and complexity of the measurement solution increase too, so be careful of that.
Learner Based:
A measurement model that captures data from training participants at two distinct points during the learning process. The first point is directly after the learning intervention (Post Event) where the main measurement focus is on Kirkpatrick’s Level I – and Level 2 to gauge satisfaction and learning effectiveness. Because there is a high response rate to these data instruments it is also critical to capture indicators for advanced levels of learning such as Level 3 – Job Impact, Level 4- Business Results and Level 5 ROI. These indicators are in effect forecasting or predicting the future impact the training will have on the participant and the organization.
A second data collection point is a follow up survey conducted a period of time after the participant has been back on the job. This survey is meant to true up the forecast and predictive indicators of Levels 3, 4 and 5 by gathering more realistic estimates now that the participant is back on the job.
The approach is low cost if one leverages standard data collection instruments across their training and utilizes technology and automation to capture, process and report the collected data. Thus it can be used for all of your training, each time a participant takes a class to yield continuous measurements.
Manager-Based:
This method has the same data collection points as the learner-based solution but adds a manager-based dimension. The manager of the participant attending training is another important data point. They can be sent an evaluation instrument timed when the participant receives a follow-up. The manager survey focuses on Levels 3, 4 and 5 of the Kirkpatrick and Phillips models therefore getting estimates surrounding job impact, business results and ROI from the manager’s perspective. The manager survey also asks ‘support’ type questions to understand the on-the-job environment where the participant applied the training.
Due to the increased effort it takes to conduct and analyze manager surveys the cost and time to measure at this level is higher than the Learner-Based approach. But, with automation and technology to facilitate the dissemination, collection, processing, and reporting of the data, the cost and time can be minimal. The result is that it could be used on a continuous basis for every training event a participant attends. More realistically, it will be used on a periodic basis for more strategic programs where manager data is more relevant.
Analyst-Based:
This approach uses significantly more comprehensive post event, follow up and manager surveys it also uses other analytical tactics that go beyond surveying. For example to analytically measure Level 2 – learning effectiveness a detailed test is designed and administered to participants. Due to the time commitment of conducting a significantly detailed data collection and analytical exercise the Analyst-Based approach is only used for about 5% of all training programs in the organization. Typically these programs are the more strategic or visible and have the budget to afford a more costly and time-consuming measurement exercise.
See Exhibit Four for an visual of the learner, manager and analyst based methodology.
Concluding Thoughts:
Activity measures are necessary to manage a training business and will tell you how much you trained. But, the workplace has changed and the demand for more comprehensive measurement tools heightens the need for better use of performance metrics that tell you how well you trained.
Through appropriate methodology that is made scaleable, practical, and replicable leveraging technology, organizations can provide performance metrics in a cost effective manner. Organizations should always keep in mind that you do not have to have the most perfectly accurate quantitative metrics in order for them to be worthwhile. In fact, a recent study published in Harvard Business Review (May 2003) found that senior managers make decisions on other instinctive factors not the highly accurate and highly costly data they are provided from highly paid number crunchers. Use reasonable data based on indicators that are meant to predict and estimate your key performance metrics. This will save you significant cost, significant time, and accomplish the objective of providing performance measures to management so they have the right metrics for their information decision-making.
About the Author:
Jeffrey A. Berk
Vice President, Products and Strategy
KnowledgeAdvisors jberk@knowledgeadvisors.com
+1 312 423 8599
Jeffrey A. Berk is Vice President of Products and Strategy for KnowledgeAdvisors. KnowledgeAdvisors is a corporate learning business intelligence firm that helps organizations gain the knowledge to improve performance, better educate its workforce and reduce costs across the enterprise. Its proprietary measurement systems and benchmarking expertise help companies more successfully implement e-learning strategies and better manage corporate learning investments.
In his current role, Berk oversees the operations and strategic direction for KnowledgeAdvisors.
Prior to joining KnowledgeAdvisors Berk worked for nine years with Andersen. While with Andersen he was the Manager of Benchmarking Services. In that role, Berk wrote the Andersen Benchmarking Methodology. He also spearheaded the development of quantitative and qualitative benchmarking tools to measure the effectiveness of client business processes.
Berk also teaches a graduate school course on continuous improvement through business process benchmarking at Loyola University in Chicago and Northwestern University in Evanston, IL. He designed and developed the course curriculum.
He has participated in analysis, integration and measurement engagements in the U.S., Europe and Australia. He has spoken on these topics and has been quoted and published in periodicals such as T+D Magazine, Training Magazine, CLO Magazine, Workforce Magazine, Fortune, IndustryWeek, and The CPA Journal.
Berk, a CPA, began his professional career first working as an auditor before moving to consulting at Andersen then KnowledgeAdvisors. He has an MBA from the University of Chicago and degrees in business and accounting from the University of Kansas.
Table One: Activity Measures
1) Fill rates by class, course, program, curricula, location, learning methodology
2) Student Cancellation rates by class, course, program, curricula, location, learning methodology
3) Min, Max, Average participants enrollments in class, course, program, curricula, location, learning methodology
4) Percent enrolled to attended ratio by class, course, program, curricula, location, learning methodology
5) Student Activity per class, course, program, curricula, location, learning methodology (% students attended, % students no show, % students cancelled)
6) Class cancellation rates (% cancelled)
7) Completion rates by learning methodology (% of elearning offered vs. completed, % of ILT offered vs. completed)
8) Instructor Utilization Rates (available to used ratio)
9) Location Utilization Rates (available to used ratio)
10) Courseware Utilization Rates (available to used ratio)
11) Courseware Aging Report (shows % of courseware that is <1 yr old, 1-2 yrs old, etc)
12) Student profile by class, course, program, curricula, location (% sr exec, % mgr, % staff)
13) Response rates to evaluations (Post event, follow up, manager, by course, class, location, program, curricula, learning methodology)
14) Financial Summary (cost per course, cost per student, cost/benefit ratio)
Table Two: Performance Measures
1)Instructor Performance
2)Courseware quality
3)Facility conduciveness to learning
4)Training vendor customer service ratings
5)Online delivery quality
6)Effectiveness of on the job support tools
7)Learning effectiveness (knowledge gain)
8)Percent of training applied to job
9)Training time to job impact
10)Barriers/Enablers of training use on the job
11)Percent of time training skills used on the job
12)Criticality of training to the job
13)Estimated change in business performance isolated to training
14)Trainings value as an investment in time/money
15)Return on training investment