Learning Analytics Research Uncovers the Current State of Learning Measurement
T&D Magazine, June 2004

In recent months significant attention has been focused on the need for better measurement and accountability for training. The term learning analytics evolved that described the set of activities an organization does that helps it understand how to better train and develop employees and customers.

As a result of the importance placed on this topic, KnowledgeAdvisors, a business intelligence software company that helps organizations measure and manage their learning investments (www.knowledgeadvisors.com) conducted research to explore the current state of learning analytics, best practices and challenges for the future. In the Fall of 2003 over 100 organizations responded to an in depth quantitative survey exploring key elements of learning analytics:technology, process, stakeholders and strategy.

KnowledgeAdvisors then tabulated the results from all respondents and has produced a final report summarizing its findings.The report includes over twenty key performance indicators quantifying where the industry is at today regarding analytics. In addition, commentary and analysis on the best practices and future challenges are noted.

This article summarizes the results of this research and highlights some of the more urgent calls to action.

Technology Leverage
Summary Findings
  • Most training organizations collect their evaluation data using paper and/or online methods. Other popular methods include email and interviews.
  • A majority of training providers store collected data in centralized databases.
  • The most commonly used database type is a spreadsheet application. Other popular applications include enterprise relational databases and departmental relational databases.
  • Overall, training organizations feel that they leverage technology and automation at a moderate level for data aggregation and data collection. However, organizations overall feel that their data reporting is slightly below a moderate level and data filtering is their weakest element.
  • When importing data to learning analytics tools from "feeder" systems, training organizations use Learning Management systems and Online Evaluation tools most often.
Call to Action
  • Technology should be sought for automation of the measurement process when collecting, process, storing and reporting data. This will lessen the time readying the data for use, enhancing the metrics for decision-making purposes.
  • Like a learning management system (LMS) three years ago, learning analytics systems are ‘nice to have’ today but within the next 12 to 18 months will be a necessity to ensure learning investments are monitored appropriately.
  • When looking at technologies that automate the learning analytics process, ensure the technology is based on industry-accepted methodology (ex. Kirkpatrick learning levels) and can easily provide reporting that is user friendly and flexible.
  • Integrating the technology with an LMS, HRIS, or ERP is important. It can ensure that the data between the systems is consistent. It also saves time and money on system administration.
Process Optimization
Summary Findings
  • Most organizations formally measure over 75% of their training events.
  • A majority of training providers do not have a standard set of key performance indicators that they measure and monitor regularly.
  • Nearly 80% of participants reported they use end of class questionnaires to survey learners on their training all the time. Post-Tests, Skills/Needs Assessments and Instructor questionnaires are used frequently or all the time by at least 30% of respondents.
  • With respect to training budgets, most training providers have less than 5% available for learning analytics.
  • Few organizations have 2 or more full-time equivalent resources to focus on learning analytics. Most have 1 or none.
  • Resources dedicated to learning analytics spend an average of only 20% of their time performing data analysis.
Call to Action
  • A best practice is to derive a standard set of key performance indicators for your learning organization that are monitored and measured on a regular basis.
  • Leverage technology and standard data collection instruments to conduct more comprehensive on the job analysis such as collecting data from learners and supervisors 2 or 3 months post training.This will allow for more robust impact data.
  • Templatize the process of linking business results to training. Use best practices such as estimation, isolation, and adjustment techniques taught by experts like Dr. Jack Phillips to make the linkage between business results and training more feasible.
  • Although studies have shown that spending 3 to 5% of your training budget on measurement is the norm, a solid process with attention to technology leverage and practical solutions can probably place a robust analytics system in place for 1 to 3% of the training budget.
  • Without standards and technology the process of measurement is labor intensive. Those that highly leveraged a robust analytics tool spend significantly less administrative time on gathering data and more time using it to make decisions and show value to their stakeholders.
Stakeholder Needs
Summary Findings
  • Training providers feel that their clients hold a significant value to performance analysis reports (job and business impact from training and ROI of training). They feel that tactical reports, aggregate reports and executive reports hold only moderate value to their clients.
  • Providers also reported that their performance analysis and executive reporting tools have slightly below average performance. Aggregate reports and tactical reports perform slightly above average.
  • Learning analytics results are communicated to stakeholders most commonly by periodic electronic/paper management reports and periodic meetings. Another popular communication method is to provide stakeholders direct access via web reporting interfaces/tools.
  • Most learning measurement outcomes are available to stakeholders on a request basis or monthly basis. Many organizations make the information available quarterly while others have it available on a continuous/real time/daily basis. Others have semi-annual/annual availability only.
  • Results of learning evaluation processes are most frequently shared with training staff and managers. Senior Managers, Training Executives and Business Units are other common recipients of this type of data.
  • Organizations provide the results of their learning analytics for many reasons. The most popular reason is to showcase the training's value to the organization. Another common reason is to indicate the quality of the training services provided. Additional reasons include because stakeholders request it or to justify large expenditures.
Calls to Action
  • The largest gap today is in conducting job and business impact analysis and ROI analysis. These are the items that matter most to stakeholders. Leveraging technology and industry accepted methodologies and guiding principles can make reporting on this type of data practical, scaleable, and replicable.
  • Communications to stakeholders is done periodically. Strive to allow stakeholders with real-time information that they can get from a self-service online reporting module. Or, use technology to automatically ‘push’ the key reports to various stakeholders on a regular basis.
  • Most organizations don’t show results unless asked to do so. Be proactive. Generate a balanced scorecard and show it to management during the budget year, not when it is over.
Analytics Strategy Summary Findings
  • Only 30% of respondents evaluate their training above Level 3 - Behavior/Job Impact. 50% of respondents evaluate training to Level 2 or Level 3, and 20% evaluate their training at Level 1 only
  • Only 62% of the respondents use benchmarking to compare their learning analytics results with that of other training organizations.
  • 80% of participants use a standard set of questions asked to all learners across all learning events as part of their measurement strategy. 49% use a core set of key performance indicators balanced across the Levels of Learning. 31% implemented built-in predictive indicators of advanced measures on their end of class evaluations, and only 14% use analytics models that are scaleable and replicable.
  • With respect to the levels of accuracy of analytics, a majority of respondents require reasonable quantitative/qualitative indicators to be considered useful for making business decisions. 11% require periodic 'word of mouth' statements, and the remaining 11% require highly precise, statistically valid measures.
  • To obtain a higher level of accuracy, 4% feel they can have as many additional resources as needed, and 18% feel they can get no additional resources to improve their data accuracy. 42% of organizations expect that they could obtain very little additional resources to increase accuracy, and 36% feel they could obtain moderate additional resources.
Call to Action
  • Because the greatest demand for analytics is above Level III, organizations should strive to create the right processes and use the most efficient and effective technologies to make Level III, IV and ROI analysis practical. A good strategy will leverage predictors of all learning levels at various points in the learning process for all learning events, not just a small percentage.
  • Benchmarking motivates by example. More organizations should strive to leverage standards that can be customized and that link to industry accepted measurement models. In doing so such instruments can be benchmarked against a normative database so that the organization can get a comparison both internally and externally.
  • Practicality is a best practice. For 95% of your learning events, gathering indicators of satisfaction, effectiveness, impact, results and ROI can be done using forecasting and predictions. This ‘roughly reasonable’ data is felt to be acceptable for decision-making and showcasing value to stakeholders. Leverage technology to gather, store, process and report these indicators so they can be delivered in a timely manner when decisions need to be made. Reserve the remaining 5% for a more detailed analysis for those high cost, visible programs. The key is practical, scaleable, and replicable!
Key Takeaways:
The following items represent some of the more poignant points from the research.
  • 81% of resources for measurement today are tied up in administrative activities (collection, aggregation, filtering, reporting) leaving less than 20% for analysis and decision-making.Technology reverses that ratio – 80% on analysis and 20% on administration.
  • Only 34% of respondent organizations had a formal set of learning key performance indicators that they measure and monitor on a regular basis. A standard set of key performance indicators should be created. For example, a balanced set of measures should include components of satisfaction, learning effectiveness, job impact, business impact, and ROI.
  • The largest gap between current analytic performance and importance is in performance analysis Trying to tie learning to job impact, business results and ROI.Showcasing the value of training is where significant improvement is needed.
  • Only 30% of respondents benchmark training results against external organizations. Benchmarking is a tool that motivates by example. Internal benchmarking is a great first step by external benchmarking provides a great reference point outside your organization.
  • 78% of respondents felt reasonable qualitative and quantitative indicators was acceptable as opposed to highly statistical measurements. Independent studies have shown that there is an enormously high cost to data accuracy versus obtaining a reasonable indicator. Organizations should balance when to take the extra time and money to yield highly statistical results. Given how executives make decisions, often times reasonable data provided in a more-timely manner outweighs data with more precision delivered in a less timely manner.

Conclusion: The results of this groundbreaking research clearly show a strong movement toward more formal measurement of training investments. However, it solidifies the notion that organizations have very little financial, physical and human resources to spend on highly precise data that may take months to gather.

Best practice learning organizations have and will continue to deliver reasonable indicators along multiple balanced scorecard learning measurement dimensions such as learner satisfaction, learning effectiveness, job impact, business results and ROI. But, these organizations will leverage automation and technology to wrap solid industry-accepted methodology around their measurement strategy. This allows them to deliver comprehensive measurement results in to 1) continuously improve their training event after event, and 2) showcase value to stakeholders.

A small investment in technology at less than 2 to 3% of the training budget is what the best practice organizations have and will continue to do to ensure their investments are monitored and measured.

These organizations continue to leverage solid industry methodology but do so in a manner that is practical to implement within their organizations. This is a very important finding as most organizations cannot spend significant resources to do one-off measurement projects and if they could, the data is needed in a timelier manner.

The key with learning measurement is crawl-walk-run. By this, I suggest starting out small. For example, ensure your evaluations are collecting business intelligence on what matters from a continuous improvement and demonstrable value standpoint. Each time data is collected from a respondent it should be an opportunity to gather impact, results and ROI data – not just reaction data. To the extent you can leverage technology to automate the measurement process it will make the management of the metrics simplified and more powerful.

Finally, recognize that you do not have to spend significant dollars and several months deploying the right learning analytics system. Building a system from scratch could take serious time and money if you try and include all bells and whistles immediately. Start with the items of highest priority and then go from there generating short term wins along the way to keep the project moving. Best practice organizations will outsource the measurement process for the day to day data collection, storage, processing and reporting. This will enable them to focus on their core competencies while saving time and money to measure their learning events.

For a complete and complimentary copy of this research report please contact Jeffrey Berk at jberk@knowledgeadvisors.com
Jeffrey A. Berk
jberk@knowledgeadvisors.com Jeffrey A. Berk is Vice President of Products and Strategy for KnowledgeAdvisors. KnowledgeAdvisors is a corporate learning business intelligence firm that helps organizations gain the knowledge to improve human performance, better educate its workforce and reduce costs across the enterprise. Its proprietary measurement technologies and benchmarking expertise help companies more successfully measure human performance change due to training.