Identifying Tangible Business Results
CLO Magazine, March 2005
By Jeffrey Berk, KnowledgeAdvisors
In order to do this topic justice I need to start by presenting the business case as to why one should read the rest of it. Without a sense of purpose the rest is just words on paper. So the article will begin with what is being heard in the marketplace regarding the importance of linking training to real business results.
The need to link the strategy, impact and ROI of training to tangible business results topped is a key priority for training executives based on research and surveys. Training executives hear this from their funding sources all the time and it is no surprise that it has percolated to the top of the priority list.
To further justify this business case, consider what IDC stated in July of 2003 “[Providing measures on the impact of training on business performance] should become a routine offering in the arsenal of training tools employed by vendors.” This prediction is now very much reality. Sophisticated stakeholders of training be it eLearning, coaching, in-house custom designed training or traditional ILT will demand some metrics showing how training impacted tangible business results.
Nothing illustrates the prior point more than a real example. PeopleSoft Education is one of the top 15 IT training providers in the world. They have thousands of enrollments in live training per year. They are the fastest growing enterprise software training provider in the world. PeopleSoft Education saw a need in the marketplace to link their training to tangible business results. What the customer craves is the real link to key results. How does the training impact cycle time, productivity, or quality? Will the training result in fewer calls to the call center, less customizations, lower implementation costs? These are great questions to try and answer but unless there is a model that can scale it is not practical to engage in scientific-like experiments delivered months or years later.
So PeopleSoft Education adapted the Phillips ROI Process to measuring the impact of training on their customer’s business results. But being a technology company, they leveraged a learning analytics technology to predict at the end of training the impact on business results and then with the help of the same technology they true those predictions up with on the job estimates months later. All data is collected, stored, processed and reported in an automated manner.
So what are the results of this innovative approach? For starters, PeopleSoft Education published a press release on March 22, 2004 that stated “Research Shows that Training Increases Productivity by 20 Percent.” Over 13,000 evaluations had been conducted both at the end of training and 60 days later and the principles of estimating productivity gains, isolating them to training and adjusting the entire analysis downward for bias, confidence and conservatism yielded this metric. Other metrics included a 24% improvement in cycle time and a 22% improvement in quality.
Now, when PeopleSoft Education conducts training they can provide reasonable indicators to their customers on how that training linked to tangible business results. Where the client wants to drill deeper and conduct a time motion study for example there are tools in the learning analytics software to house such data points, isolate them to training and adjust them for conservatism. However, most clients are looking for a reasonable, businesslike approach to linking training to business results. Cost, time and limited personnel prevent in depth scientific analysis most of the time. Further, PeopleSoft Education can use their database, now much more significant that 13,000 evaluations, to model and predict how each course they run impacts business results on future clients. They can tweak the course to maximize impact thereby maximizing their customer’s value from training dollars.
What is Learning Analytics and How Does Impact Fit In?
The term ‘learning analytics’ was more than once in the opening section of this article. It might make sense by now to define it further and provide further detail about how linking training to tangible business results fits into the learning analytics model.
KnowledgeAdvisors defines learning analytics as “a technology that helps organizations understand how to better train and develop employees, partners, and customers.” There are four components of a comprehensive learning analytics system. They are illustrated in Figure I.
Figure I illustrate evaluation, testing and assessment, operations and impact as the four sources of learning analytics. This section of the article will discuss how each fits into a comprehensive learning analytics system.
First there is evaluation data. Never under estimate the power of this data. Rather than squander an opportunity to collect data from a participant, expert, manager, instructor or other stakeholder by only asking typical smile sheet indicators such as instructor performance, courseware quality or environment conduciveness to learning, let respondents document their predicted and perceived learning effectiveness, job impact, linkage to significant business results and even a financial ROI. Evaluation data is not always the preferred method of gathering data but it is often times the only realistic method given financial, physical and human resource constraints so definitely take advantage of this data to get some reasonable indicators of linkage to business results.
An example of how organizations use evaluation data creatively is Eaton Corporation. Eaton University, a training element of Eaton Corporation, won the Corporate University Best In Class Award (CUBIC) for best evaluation technique in 2003. Eaton University has a completely automated a process whereby their Learning Management System feeds their learning analytics system nightly via XML FTP feeds with all kinds of class details and respondent details. Evaluations are then sent out the next day to respondents for electronic data collection providing real-time business intelligence to the university to monitor satisfaction, understand knowledge transfer, predict job impact and linkage to business results, and calculate a financial ROI based on the human capital performance gains. The process then repeats itself 2 months later by automatically sending a ‘follow-up’ evaluation to the respondent and a ‘manager’ evaluation to the supervisor to ‘true-up’ in an automated and scaleable manner items like time to job impact, significant business results impacted back on the job, and recalculate the financial ROI.
Second, there is testing and assessment data. When a program has a safety, business risk, or regulatory driver a test may be reasonable. Therefore testing data should comprise a component of overall analytics, albeit a small component. The notion that a learning organization cannot achieve a link to business results (inherent in Kirkpatrick’s Level IV) without first traversing learning effectiveness (inherent in Kirkpatrick’s Level II) is misguided, harmful and counterproductive.
An example of this was a major financial institution that collected smile sheet indicators. Management requested linkage to tangible business results. So the learning organization felt that in order to get there they had to test first, achieve Level II then they would be a step closer to what management really wanted. They pulled instructors out of the classroom; they pulled course designers off of new course design and focused on the creation and delivery of tests for a significant number of classes. Not only did this create frustration within the organization but the learners expressed dissatisfaction over being tested for minor details and an overall feeling of distrust. Finally, management never understood why their request was not honored as they did not understand who Kirkpatrick was and why his Level II needed to be done before they could understand if sales training really resulted in increased sales.
Third, there is operational data. Operational data focusing on answering the question ‘how much you train’ versus the other components of analytics focus on ‘how well you train.’ Any line of business needs to monitor operations and metrics help you do that. So your learning analytics system should have some component of operational data. Operational data includes items such as enrollment rates, cancellation rates, instructor utilization rates, course or location fill rates, and evaluation response rates. Many of these metrics are derived from basic registration data on your learning typically housed in most learning management systems but poorly reported. Often organizations struggle in providing this basic data engaging their IT departments in writing numerous queries in tools like Crystal Reports that is often kluge and cumbersome for this task.
A great example of leveraging learning analytics for this type of analysis is Caterpillar Inc. Caterpillar recently won the CUBIC aware for best evaluation technique in 2004. Caterpillar’s learning management system sends their analytics system nightly files containing course details and enrollment details allowing the analytics tool to then easily calculate fill rates, response rates, enrollment rates, and utilization rates. Further, all of these rates can be filtered by various components of the learning operation (by course, by location, by instructor, by learning delivery etc.).
Finally there is impact data. This is a key component of learning analytics and is the heart of this article. A learning analytics tool should allow for the capture of impact data in multiple ways. One way we have discussed already is via evaluations. Taking the opportunity as we saw with PeopleSoft Education, to expand the evaluations to predict and estimate impact while applying the appropriate methodologies is effective, practical and scaleable for a learning operation of that magnitude. However, PeopleSoft Education and other organizations large and small, realize that certain stakeholders have or are willing to gather results oriented data to compliment or use in substitute for evaluation data.
Impact data should be carefully gathered and done in the appropriate context and scope of the learning program and the environment in which the business uses the learning. Ideally, a learning analytics tool does not simply take data from one system match it up against learning and declare impact. That can be harmful and dangerous. For example, article on sales training promoted an ROI beyond 4000%. The reason behind this was that a major competitor had gone out of business at the same time sales training was conducted so by coincidence there was a spike in sales after sales training. A learning organization can quickly lose credibility if it does not isolate the impact to training.
A learning analytics tool that allows the learning organization to engage the stakeholder in a consultative manner in gathering the pre and post metrics, funneling the root cause of change and adjusting the data for conservatism yields a more credible business impact and ROI figure. For example, a bank does IT training and a macro business result expected from it is increased productivity as defined by the VP of IT. The micro definition of this business result is time to complete a specific set of tasks. The company actually has resources set aside to sample via time studies this result and does so before and after the training. If the entire workforce is not trained they have a naturally occurring control group and place that data alongside the trained group for an additional analysis.
The learning organization is then able to input the sample results pre vs. post training and versus the control group into the analytics tool. But, rather than claim the entire productivity gain (if there was one) to the training, the consultatively walk IT managers through a root cause analysis in the analytics tool to determine if the business result net change was due to training, technology, policies and procedures, external factors etc. Finally, they place a percentage on the net analysis via the tool to adjust it downward for conservatism. The final result of what may start out as a 30% gain in productivity is that there is a 7% gain due to training, adjusted for conservatism.
The key to the aforementioned analysis is having the right tools and templates that steer the stakeholder and learning organization through the appropriate calculations. Recognizing that this depth of analysis may not be done all the time is important in realizing that you can use evaluation predications the rest of the time as opposed to smile sheeting 90% of your training and providing business result linkage on 10% of your programs.
Another important point is that the linkage to business results requires a concentrated effort to really know the macro result (examples of the major macro results include revenue, quality, business risk, productivity, cycle time, cost, quality, customer satisfaction, and employee turnover) and the micro definition (example for quality it may be order fulfillment error rates per person per day). To do this effectively, suggests that the analysis is only as good as the data and its source. Finding another system within the organization that tracks your stakeholders micro business results can be challenging. Equally challenging is integrating it with your analytics tools – an often costly and time consuming task. So, pick and chose the battles wisely when linking training to tangible business results.
Concluding ThoughtsThe Linde Corporation, a major manufacturer of welding equipment once declared, “We are very interested in measuring ROI of training and certification but I don’t know of a feasible method of doing this.” Target Corporation, a major mass merchandise retailer stated, “Having someone in training for 5 days is a huge cost to us; I can’t do that unless I know that 95% of that is job relevant.” These comments are being heard all the time when budgets are up for renewal or a new program is being proposed. Having a practical, scaleable and repeatable model to link training to business results is key in ‘selling’ the budget and the program to the stakeholder. The analysis does not have to be scientific and over-engineered but it does have to have some sense of reasonableness and credibility. As a learning organization, the fist step is to recognize like the 500 learning executives in the study I mentioned have realized; that business impact is important and you need to solve that linkage problem in some reasonable manner allowing your stakeholder to come back for more detailed analysis where appropriate.
A final concluding thought and a nice compliment to this articles’side bar is Nextel Corporation’s philosophy. They use a learning analytics tool to automatically collect, store, process, and report thousands of data points continuously on all of their training. At first, the learning organization felt a compelled not just to provide reasonable indicators of the results of their training but to dive deep with more in depth analysis. The general sense was that stakeholders would value that deep dive. The reality was that the reasonable indicators were convincing enough to show how training impacts business results. So, know your stakeholders and link training to tangible results in a way that can show value but not divert resources from your core business – training.
Jeffrey A. Berk jberk@knowledgeadvisors.comJeffrey A. Berk is Vice President of Products and Strategy for KnowledgeAdvisors. KnowledgeAdvisors is a corporate learning business intelligence firm that helps organizations gain the knowledge to improve human performance, better educate its workforce and reduce costs across the enterprise. Its proprietary measurement technologies and benchmarking expertise help companies more successfully measure human performance change due to training.