Posts Tagged ‘measure training’
Republished from ASTD (Online) OD/Leadership News, June 2006
By John Skinner
There are numerous metrics that serve specific purposes. Some of the most critical include performance metrics, operations metrics, financial metrics, and cultural metrics. As the names indicate, performance metrics deal with the actual performance of learning programs relative to a set of criteria, operations metrics describe what is going on in the organization, financial metrics catalogue the investments made, and cultural metrics tell the story about the overall organization.
How to Collect This Data
Once you understand these four categories of metrics, you have several possible methods to collect data, including surveys, interviews, focus groups, and empirical research. Depending on your particular situation, each method has its advantages and disadvantages.
Surveys are easy to scale. Items on the survey should conform to the metrics of interest-and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.
Focus groups are valuable sources of information as they allow you to dig deeper with follow-up questions. The most important thing to remember when using focus groups is structure; you should maintain a list of questions that conform with your metrics and follow it. The drawback to focus groups is that they are time intensive and are not as easy to scale.
Empirical research can serve as a goldmine of information. While this method can be time intensive, the results often are highly credible. Some examples of empirical research include control-group studies or statistically linking training to quantifiable organizational outcomes. You may need to engage an outside consultant to help with an empirical study, but this can be worth it for costly, visible, and/or strategic programs where credibility and precision in the resulting metrics are paramount.
Interpreting the Data
Once you collect the data, you must properly interpret it, aggregate it, and populate the desired metrics.
Aggregation refers to the level you choose to examine the data. For example, if you want to look at a level 1 metric, instructor performance, you can look at the data at the class level: What was the performance for this particular class? You also can look at this from the instructor level: What is this instructor’s performance overall across classes? Having a clear idea of what levels of aggregation may be needed for various metrics before data collection begins is important.
Aggregation can be thought of as filtering the data. You can only filter, or aggregate, based on certain criteria that you collect with the rest of the data. Knowing which metrics need to be aggregated and presented at what levels ahead of time can save you headaches down the road.
Frame of Reference
Even with the best metrics and data, they should not be interpreted in a vacuum. It is essential to establish context for quantitative metrics.
Benchmarking against an external data set from similar learning programs can put metrics in perspective. This information can be crucial to maintaining a human capital edge in a competitive industry. Internal benchmarks also can provide context for metrics.
Finally, the combination of internal and external benchmarks can help set goals. Goals themselves can be an important context for evaluating the performance and effectiveness of the learning organization, as well as a process check when initiating improvement plans.
As you can see, it is crucial to have a strategy in place to effectively leverage learning metrics. If you use the information presented here as a guide to setting your strategy, you will be much better positioned to adapt and scale to meet the needs of your organization and its stakeholders.
Republished from CLO Magazine, April 2006
By John Skinner
A Systematic Approach for Learning Organizations to Assess Needs, Effectiveness, Business Results, ROI and Profit Impact
There are numerous metrics that serve specific purposes. Some of the most critical include performance metrics, operations metrics, financial metrics, and cultural metrics. As the names indicate, performance metrics deal with the actual performance of learning programs relative to a set of criteria, operations metrics describe what is going on in the organization, financial metrics catalogue the investments made, and cultural metrics tell the story about the overall organization.
A constant struggle every learning organization faces is optimizing value for the investment. Optimizing is decision-making based on the past for the future. There must be a process to optimizing value and it must be built into the fabric of the learning organization to ensure it is functioning as designed.
If a process exists to not only quantify value on prior investments but forecast future value the benefits are significant and include:
Surveys are easy to scale. Items on the survey should conform to the metrics of interest-and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.
- Ensures that business results are aligned with business objectives
- Significantly reduces wasted L&D expenses
- Substantially increases productivity
- Simplifies the process of tracking actual business results
- Cost-effective approach to measuring ROI
- Enables you to link learning investments to bottom-line impact
- Data-driven approach to optimizing the impact of your L&D programs
Without a systems and processes in place to optimize value, most organizations are not reaping the full benefit of their Learning and Development (L&D) investments. More importantly, by systematically finding ways to improve impact, organizations can significantly impact earnings. For example, if an organization with $1 billion in labor cost can increase productivity by just 1%, it will increase earnings by $10 million.
Value optimization enables organizations to redirect wasted or poor-performing L&D investments into higher impact programs, which will significantly impact the bottom-line.
Case Study Highlight: The award winning Defense Acquisition University (DAU) led by CLO of the Year Frank Anderson implemented a data-driven approach to analyze value.
Over the course of three years DAU was able to significantly increase output, effectiveness and business impact without any increase in funding. This is a clear example of how an organization can systematically reduce waste and redirect investments into higher impact programs.
Human Capital Contribution Model
A systematic process to valuing learning investments is referred to as the Human Capital Contribution Model or HCCM. The model is composed of 5 core components.
Business Needs Analysis is the up-front planning process used to understand the driving business factors prompting the creation of a learning and development program.
Why is this important? Because the key to a successful learning and development intervention that maximizes impact on the business begins with a formal analysis of business needs.
What’s the process? To ensure the right L&D investment is designed, developed and delivered a needs assessment should be conducted to understand the business objectives. Once identified, a knowledge and skill assessment should be conducted to examine the gap between the current state and desired state.
What are the tools? The primary tools are needs assessments, knowledge and skills assessments, pre and post tests, and competency assessments. Using learning analytics technologies to collect, store, process and report these results, greatly facilitates the process.
What are the results? Better designed learning programs and more clearly defined and measurable business objectives that can be linked to post-learning business results.
A case study is the example of CNA Insurance. The organization was designing a new learning program. A 40 question scenario-based skill assessment was used. The net result was the skill assessment helped in the development and delivery of the training to make it more targeted to the end user.
Performance Analysis is a systematic approach to redirecting poor performing investments into programs that drive on-the-job productivity.
Why is this important? Because research has shown that nearly half (50%) of all training is wasted. In financial terms, for a typical Fortune 500 company that is more than $50 million per year, more importantly, a typical Fortune 500 company could drive another $10 million in earnings if productivity were improved by 1%.
What’s the process? To ensure waste is eliminated and productivity continually improved a systematic review of key indicators is needed to pinpoint the root cause of poor performing investments. Performance analysis focuses on where the investment occurs (Which programs? Which courses? Which vendors? What customers or lines of business?). It also analyzes key attributes of activity and performance data including attendance rates, learning effectiveness, job impact, and value indicators.
What are the tools? Data sources for this analysis include LMS (learning management system) data and evaluation data. Using learning analytics technologies to collect, store, process and report these results greatly facilitates the process.
What are the results? Better information for decision-making. Future allocations of L&D resources to more value-added programs, courses, vendors, and clients or lines of will significantly impact the bottom line.
Case Study Highlight: Nextel Communications, Inc. was conducting sales training. The company hired a third party to assist in delivery of it. Within the first days of the rollout the company was able to quickly review the results of the third party instructors. The results illustrated that certain instructors were poorly performing. The results of the above helped to eliminate waste and improve quality.
Business Result Analysis is a systematic approach to connecting L&D investments to a balanced set of actual results.
Why is this important? Because you can’t manage what you don’t measure. By systematically analyzing the link between learning and business results, one can identify areas for improvement and help drive better results. In addition, by analyzing business results, validation for ongoing investments can be made.
What’s the process? HCCMTM is based on a balanced approach and being as closely aligned to financial statements as possible. After all senior management performance is almost always based on financial results.
The recommended balanced set of business results most closely aligned with financial statements are:
- Revenue
- Profitability
- Productivity
This approach applies to both the public and private sector and is recommended for all major learning programs including corporate universities, leadership programs and business unit programs. When appropriate, it is also recommended that additional business results be tracked such as quality, cycle-time, employee loyalty, customer loyalty and risk mitigation.
What are the tools? Data sources for this analysis include ERP (enterprise resource planning data), financial and cost accounting data, customer satisfaction data, error rate data, CRM (customer relationship management) data, and HRIS (human resource information systems) data. Learning analytics technologies provide L&D managers with templates and wizards to analyze the key business results but is flexible to provide for customized business results analysis and then displays the analysis on interpretive and graphical scorecards and dashboards.
What are the results? A thorough and timely understanding of organizational impact. L&D becomes advisors for improved performance.
Case Study Highlight: PeopleSoft, Inc. (prior to the Oracle merger) needed to tie the investment in training to a core set of business results. The company used both evaluation data where business results were linked to training both at the conclusion of training and months later and also an automated template to link actual business results with training. The outcome was a quantification of results that showed an improvement of 20% in productivity with the training.
ROI Analysis is a process-based approach to determining the financial return on L&D investments given cost considerations.
Why is this important? Because investments should provide more positive financial value back to the organization than the resources consumed in the original investment. It is also a commonly used financial analysis to validate investment decisions.
What’s the process? To ensure L&D management focus on benefit vs. cost leveraging methodologies from experts such as Dr. Jack J. Phillips’ ROI Process helps provide a framework for ROI. The process of collecting, storing, processing, and reporting data that create a monetary cost and benefit from training are inputs to the ROI process and then isolating the benefit to training and adjusting it for bias and conservatism is part of the Guiding Principles of the ROI Process.
What are the tools? Data sources for this analysis include ERP (enterprise resource planning data), financial and cost accounting data, customer satisfaction data, error rate data, CRM (customer relationship management) data, and HRIS (human resource information systems) data. Leverage analytics technologies to wrap automation, templates and wizards around the Phillips ROI Process to make the administrative burden of ROI feasible and practical when there are limitations on resources to apply to ROI analysis.
What are the results? A credible and reliable ROI expressed as a benefit to cost ratio that when compared by learning delivery, program, vendor, client or line of business will help the L&D manager make better decisions from a dual dimensional perspective (benefit and cost). It will also validate the investment (or invalidate it in the case of a negative ROI).
Case Study Highlight: New Horizons Computer Learning Centers, Inc. was assisting a client in technology training. Management was very interested in the financial ROI.
The key focus was on estimating the change in performance before versus after the training and isolating it to training and then adjusting it for confidence and conservatism. The cost of the training was also conservative by design.
The outcome was an ROI that was positive and produced future economic value as seen through a 3.43 payback period.
Profit Impact Analysis is a data driven planning and reporting process that helps to optimize the impact of L&D investments and to connect learning to financial statements.
Why is this important? Because L&D expense is part of the calculation, these profit measures are arguably the most meaningful ways to analyze past and future impact of learning on earnings.
What’s the process? If learning intervention is effective, it will ultimately impact profit in a positive way. The financial goal of a corporate university is to increase the profit contribution of its people. That profit contribution can best be measured by taking revenue less labor costs and L&D expense, which is called The Human Capital Contribution Profit. A corporate university will be most successful when the Human Capital Contribution Profit is growing faster then the revenue, which is measured by calculating the Human Capital Contribution Margin (Human Capital Contribution Profit divided by Revenue).
What are the tools? Data sources for this analysis include financial and accounting systems. Learning analytics technologies can provide templates to guide a manager through the inputs and structured analysis to trend and track the actual to projected results.
What are the results? A sensitivity analysis tool that can help plan investments better and optimize L&D impact. It also enables learning professionals to discuss financial impact with business executives.
Case Study Highlight: KnowledgeAdvisors, Inc. a learning analytics technology company reviewed the profit impact from a major investment in a new technology platform and analyzed the training effect on the investment. The company took a financial approach to analyze this important learning and development investment.
Projections were made for the 2nd and 3rd years after the training as management set out to prove that although the training investment would be significant it would pay off in the form of future earnings that directly impact the bottom line. The second year projection using a profit impact analysis approach was a 6% improvement in profit margin. This was because the new technology platform was projected to be brought online quicker and managed more effectively as a result of the training (the human capital contribution effect) and ultimately help accelerate future sales by 5%. The 3rd year was projected to increase profitability by 3% with a 3% increase in revenue due to the training investment.
Conclusion
Optimizing value in learning and development investments is not easy. However, it is not impossible. The right systematic process, technology and templates can facilitate the analysis.
In today’s world of limited money, time and personnel we face an era of accountability for all that we do. If we are to be the fiduciary of millions of dollars for learning and development, management should expect nothing less than 1) a reasonable analysis on the business needs that drive the investment, 2) the manner in which we will optimize the performance of the investment so as to lower our risk of wasted training, 3) the linkage of the investment to key business results like revenue, profitability and productivity, 4) a reasonable ROI calculation based on the isolated and adjusted benefit from training, and 5) the projected impact on future earnings.
Based on observation in other lines of business such as sales, marketing, production planning, warranty and claims etc. these analysis have been done for years for equal strategic, visible or costly investments. Learning and development organizations should leverage the best practices of other lines of business and derive roughly reasonable ways to optimize value. Business is not about precision and perfection it’s about how to provide reasonable information in a timely manner to make better decisions. Keeping this in mind will go a long way in demonstrating accountability while illustrating how the investments will generate the most for the stakeholders.
Republished from: Measurement, Evaluation and ROI News, March/April 2006
By Jeffrey Berk
In today’s world of learning analytics it is easy to get caught up in a whirlwind of ROI, linkage to business results and bottom line impact terminologies. However, what about the basics of evaluation? How can 21st Century technologies make this simple yet powerful process more practical and scaleable within a learning and development (L&D) organization where they just need to see the results of a Level 1 Reaction survey or a Level 2 test? This article will take us back to the basics of evaluation by exploring the core components of the process of evaluation and offer tips and examples of how to make it easier where there are limited financial, physical and human resources to accomplish the basic process of evaluation.
Data collection:
There are a host of inexpensive web-based data collection tools. These tools are easy to use and allow you to collect data from learners, instructors, managers, and any stakeholder at any point in time. Leveraging the internet to collect data saves costs of paper processing. It is worth investigating and leveraging where practical and feasible. Response rates are not dramatically different (65% online / email vs. 85% paper based on research). Even if paper is necessary, scanning technologies can easily scan quantitative and qualitative data into databases for centralized data storage and processing.
Technologies specifically designed for learning evaluation will provide standard evaluation tools to take advantage of benchmarking. They will also have multiple ways to get data into the storage component (discussed below) such as online, email, data entry, and data import.
Data Storage When it comes to data storage, I don’t mean that metal file cabinet in the basement closet of your office building that so many end of class evaluations go to as their final resting home. Simple tools like spreadsheet applications can be inexpensive and powerful technologies where data collected online can be dumped for efficient, centralized data processing. More sophisticated solutions include using local relational databases such as Access that is more powerful than Excel or using enterprise relational databases such as SQL or Oracle that can span the organization easier. Data storage in technology tools is central, secure, and makes processing easier than that basement file cabinet.
Technologies specifically designed for learning evaluation will have centralized data storage to warehouse your evaluation data, setting you up for success in data processing and reporting.
Data Processing
Data processing is probably the most time intensive piece in all of learning measurement. The key here is to have flexible tools that allow for the building of queries that can then be automated or standardized. OLAP tools such as Cognos or Microsoft Analysis Services are very powerful for querying large amounts of data. However, a strong word of caution. Do not throw technology into the hands of the functional users untrained on learning measurement. Allowing a line of business person to write their own OLAP query that compares learning data to business data can be counter productive if not dangerous if they do not know what they are doing.
Technologies specifically designed for learning evaluation will have a very user friendly front end to allow a learning consultant to query their Level 1 evaluation data by attributes of the training (instructor, course, location, date, delivery) and by attribute of the trainee (business unit, client, job function etc.). It is just a matter of making easy selections in a wizard-like tool and the output is exactly what you need.
Data Reporting
In most organizations there is a need for 4 primary types of reports. Your measurement solution needs to be able to provide reports for each. These reports include tactical, aggregate, executive, and value analysis.
Let’s start with tactical reports. These are for the staff personnel in the training department that need to manage the learning investments on a day to day basis. Examples of tactical reports include class evaluation summaries, respondent qualitative verbatim, and the actual evaluations themselves. These are used to spot problems before they can get any bigger.
Next are the aggregate reports. Middle managers in a learning group use these reports. They aggregate tactical data for monitoring and quality control. For example the person responsible for managing the sales curricula needs to quickly view all the course titles and determine which are most effective. The person managing the eLearning content needs to easily see how effective it is compared to traditional instructor-led training. These managers need to see tactical data rolled up and they need to be able to filter it down too. Reports for these folks should do this.
Then executives in the learning group need reports. They need to see comparisons internally and externally. For example how did employees in each line of business that fund the training see the training impact their jobs? How did the key indicators of training compare to external benchmarks? And what exceptions occur on a day to day basis? Executives need summary, exception-based data that is comparable internally and externally.
Finally there is value analysis. This is packaging it all together. It is your balanced scorecard that you can present to management when the budget is up for renewal or you need to fund a new program. It’s the measures you have an appointment with CEO to review on a quarterly basis cut by eLearning vs. ILT so the CEO can see how that eLearning investment is doing. Value analysis sums it all up. It presents all the measures in an easy-to-generate and easy-to-interpret score-card. Value analysis allows a chief learning officer to sit down with a C level person or a line of business manager to cover how training impacted the job, how human capital was improved through training, and yes, the financial return on training in hard and soft dollars. Metrics such as benefit to cost ratio, payback period and ROI percentage are in the value analysis.
Technologies specifically designed for learning evaluation will have canned reports that output to these types of analysis. They will allow for easy configuration of those reports to ensure the data in the report is relevant to the need. Finally they will have multiple output formats such as PDF, web, printer, email, and excel.
Conclusion
Basic evaluation does not need to take considerable money, time and people. In today’s world of technology, even a small learning department can leverage technologies specifically designed for learning evaluation to process thousands of evaluations for a cost that is likely to not even be a rounding error in their learning and development budget.
It is well worth the time to explore a learning analytics solution to outsource the basic administration of evaluation. This will allow the learning organization to focus on value and continuous improvement
About the Author
Jeffrey Berk is the Vice President of Products and Strategy for KnowledgeAdvisors, a learning analytics technology company. KnowledgeAdvisors helps organizations improve the efficiency of the basic evaluation process through easy-to-use technology tools specifically designed for learning evaluation.
Metrics that Matter® Learning Analytics at a Major Financial Services Company
Company Overview: The organization is in the financial services industry and is one of the top 25 largest banks in the United States. The company employs over 15,000 people and has over 1.5 million customers. Continued
Cable Television Company Uses Metrics that Matter® to Quantify the ROI of a Major Leadership Initiative
Company Overview: This cable television company employs approximately 1,700 people in one of its locations. The service area covers most of Central North Carolina and has over 500,000 subscribers of digital video, high speed data, and voice over internet protocol (VOIP) digital phone service. Continued
Pharmaceutical Company Uses Metrics that Matter® to Validate the Value of Its Leadership Program
Company Overview: The pharmaceutical firm recognized that cultivating leaders was important to the success of its organization. As a result, the company established a leadership program to build organizational bench strength and groom future leaders. Continued
