Posts Tagged ‘Learning Stakeholders’
Republished from HR.com, March 2004
By Jeffrey Berk
In the past 12 months a tidal wave of information has been generated by the learning industry on the need and power of learning analytics. Learning Analytics is the set of activities a learning organization does that helps it understand how to better train and develop employees and customers.
A survey was prepared and data was collected. Over 100 training departments responded from a variety of industries. Now the results are in.
The results are organized around the key constructs we used in our survey. Looking at technology, process, stakeholders, and strategy.
Each week Hr.com will provide a look into the current trends and future state of learning analytics for each of these key constructs.
Technology
This article takes an in depth look at how technology can be leveraged make the learning analytics process more automated and streamlined.
Current Practices from the research:
The summary points below articulate the way organizations leverage technology in their learning analytics process today.
- Most organizations still leverage paper as the primary data collection technique when gathering metrics on learning investments.
About one quarter of the organizations do not store learning measurement data in a centralized database. - Two thirds of organizations use spreadsheet applications as their database of choice when storing metrics on learning measurement data.
- Just over one third of organizations have the capability to compare learning measurement data by learning delivery method
- Little or no automation is leveraged by respondent organizations to filter and query the metrics.
- Just over half of the respondents have the capability to import data from a learning management system into their evaluation and measurement systems.
Best Practice Improvement Opportunities:
- The following points highlight what best practice organizations are doing to optimize how they leverage technology for their learning analytics process.
- Leverage technology throughout all elements of the measurement process. There are four primary elements of measurement: data collection, data storage, data processing and data reporting. Technology should make each of these more efficient. The result is minimized administration time preparing metrics and maximized time doing something with them.
- Use a centralized database to store and query your data. Powerful tools (i.e. OLAP) tools exist to query large amounts of data. But, be careful. Technology is a mere enabler. Throwing a powerful query engine at a novice user can be dangerous. You’ll need to build the right front end interfaces and back end reporting around your query engine to maximize its usage.
- Technology should be leveraged to ‘tag’ all the data elements you want to report.The most practical ways to manage a learning organization are to slice your data by learning provider, location, business unit, class, course, curricula, learning delivery, and instructor.
- Learning analytics systems should ‘talk’ to other systems such as an LMS. Using code like XML can link the systems and minimize redundant tasks and mitigate risk of error.
Conclusion:
Technology is extremely important to optimizing your analytics solution. A small investment in technology can pay enormous dividends short and long term. Technology exists today that is very affordable (usually less than 2% of your training budget) to comprehensively streamline your measurement solution.
Process
This article takes an in depth look at the key inputs, activities, and outputs that comprise the learning analytics process.
Current Practices from the research:
The summary points below articulate the elements of today’s learning analytics processes.
- Just under two-thirds of responding organizations measure 76 to 100% of their training in a formal manner.
- Only one third of responding organizations have a standard set of key performance indicators that they regularly measure and monitor on their training investments.
- Most all organizations use an end of class ‘smile sheet’ instrument to evaluate training but less than 30% use on the job data collection instruments that go to the participant of the training and their direct supervisor.
- Over half of responding organizations have budgeted 5% or less for learning measurement. Nearly one quarter have no budget at all for learning measurement.
- Over one third of responding organizations have no full time equivalent resources to focus on learning measurement within their organizations. Most have one or two.
Over 80% of a respondent organizations resources for measurement is consumed in administrative aspects of measurement leaving less than 20% to actually improve training from the metrics or use the metrics to show value to stakeholders.
Best Practice Improvement Opportunities:
The following points highlight what best practice organizations are doing to optimize their learning analytics processes.
- Leverage a standard set of key performance indicators linked to industry accepted learning measurement models (i.e. Kirkpatrick Learning Levels) that are practical and scaleable indicators and predictors. Collect this data for each class and then aggregate it, and slice and dice it on a formal performance scorecard.
- Leverage the use of impact data instruments not solely smile sheets. Impact instruments look at job impact, business results and ROI when the participant is back on the job and in the best position to provide feedback on these more critical elements of measurement. Sending a similar instrument to the managers also reinforce your measurements.
- If you use technology to automate the collection, storage, processing and reporting of the resulting data then the process is optimized.
Dissect each step of the measurement model. Look at how you can leverage automation and technology and streamline steps for collecting data, storing data, processing data, and reporting data. Best practice companies that do this actually spend 20% of their time on admin activities and 80% on using the metrics to improve training and demonstrating value to stakeholders.
Conclusion:
Appropriate processes that identify measurement strategic objectives and then measure against those using technology to do the heavy lifting is critical to success. Equally critical is building processes wrapped around not only technology but industry accepted methodology for learning measurement. This is important because it helps with change management and credibility concerns.
Stakeholders
This article takes an in depth look at the critical consumers/customers/users of learning analytics reports and data.
Current Practices from the research:
The summary points below articulate the elements of today’s learning analytics stakeholders.
- Performance analysis (showing impact, results and ROI) is the single item where the largest gap exists whereby learning organizations are poor performers in this area but stakeholders feel it is extremely important.
- Over two thirds of communications to stakeholders are periodic meetings and conversations or a management report.
- Nearly one third of communications to stakeholders are done on an as requested basis which is often a reactive communication style.
- The largest consumers of learning analytic data are training staff and management.
- The top reason responding organizations cited for measuring their training was to showcase the value to the organization.
Best Practice Improvement Opportunities:
The following points highlight what best practice organizations are doing to optimize their communications with stakeholders.
- Find practical, scaleable, and replicable ways to do performance analysis (impact, results and ROI metrics) so that stakeholders get timely information that is significantly more salient to the reasons why they budgeted for the training.
- Leverage proactive communications to stakeholders such as automated ‘push reporting’ and access to online measurement reporting systems to allow stakeholders with metric data in a self-service model.
- Showing measurement data to stakeholders using the right metrics (impact, results, and ROI) is a great way to expand the use of the metric data beyond the walls of the learning department.
- Because the most significant factor in going through the process of learning measurement is showing value to stakeholders, ensure what you show to illustrate value is important to the stakeholder. Focus on linking training to their business objectives, illustrating the behavior changes made on the job, and quantifying the financial ROI on the investment.
Conclusion:
This study and other independent research shows that a major priority of learning organizations is to prove the value of training. In a world where training must compete with marketing, sales, and other department dollars, providing the value of how those dollars will be used is essential. Deploy practical approaches to gathering and presenting this data to the stakeholder to convince them that the training did have value and that future investments are well worth it.
Conclusion:
Appropriate processes that identify measurement strategic objectives and then measure against those using technology to do the heavy lifting is critical to success. Equally critical is building processes wrapped around not only technology but industry accepted methodology for learning measurement. This is important because it helps with change management and credibility concerns.
Strategy
This article takes an in depth look at how technology can be leveraged make the learning analytics process more automated and streamlined.
Current Practices from the research:
The summary points below articulate the way organizations derive learning analytics strategy today.
Only 18% of responding organizations measure the ROI on their training investments
Only 30% benchmark training investments externally by industry, job function etc.
Only 14% of responding organizations have analytics tools that are scaleable, and replicable to use for day-to-day measurement.
Nearly 80% of responding organizations feel that reasonable quantitative and qualitative data is required by stakeholders as opposed to the need to get highly precise metrics that may take more time and money to generate.
Nearly two thirds of respondent organizations have very little or no additional funding to generate highly precise metrics as opposed to reasonable indicators.
Best Practice Improvement Opportunities:
The following points highlight what best practice organizations are doing to optimize their learning analytics strategies.
- Deploy a measurement strategy that is based on industry accepted measurement methodology but is practical to implement within the organization leveraging technology to increase practicality. In this way ROI, impact and result driven data can be captured and reported more efficiently and effectively.
- Training investments should be benchmarked internally and externally so the organization can use it as a constructive tool to set goals and objectives, continuously improve and motivate by example.
- Take advantage of reasonable indicators and predictors versus feeling the obligation to deliver highly precise data that may consume large amounts of resources and not be timely enough for decision-making purposes. The reasonable indicators, however, should still link to industry accepted methodology for measurement but do so in a practical manner.
- Using reasonable indicators coupled with technology will accomplish your learning analytics strategy at a cost that is likely to be less than 2% of the training budget. Independent studies have shown that organizations should be investing 3-5% in measurement of learning.
Conclusion:
Strategy creation begins with the end in mind. How does the learning measurement strategy tie to the business objectives? How will you ensure that learning programs do tie to business objectives in practice? These are key questions that need to be addressed when creating a measurement strategy. Finally, strategy needs to be attainable from an execution perspective. If your measurement processes and technologies cannot accomplish the measurement strategy there is a disconnect that needs to be addressed.
