Posts Tagged ‘learning measurment technology’
Robert O. Brinkerhoff, Pioneer of the Success Case Evaluation Method®, Joins KnowledgeAdvisors Advisory Board
Chicago, IL [January 8, 2008] – KnowledgeAdvisors announced today that Robert O. Brinkerhoff, Ed.D. has joined the company’s advisory board and will collaborate with its staff to integrate the Success Case Evaluation Method® into Metrics that Matter® analytics technology. Continued
KnowledgeAdvisors Announces Completion of Major Expansion of Human Capital Analytics Toolset
Chicago, IL [September 12, 2007] – KnowledgeAdvisors announced today that the company has greatly expanded its Human Capital Analytics toolset within Metrics that Matter® in order to provide learning & development and human resources professionals practical, scalable and replicable metrics on four key areas of talent management processes. Continued
Republished from CLO Magazine, September 2007
By Jeffrey Berk
This article is about understanding what you don’t know. When you endeavor to try to know what you don’t know, we call that a readiness assessment. We’ll spend most of the article discussing readiness assessments from a learning and development perspective as it relates to measurement, evaluation and analytics. However we devote some space near the end of the article to discuss the larger talent management and human capital elements of analytics. Continued
KnowledgeAdvisors Partners with Applied Skills & Knowledge to Automate Vast Competency Library
Chicago, IL [June 20, 2007] – KnowledgeAdvisors announced today that it has entered an agreement with Applied Skills & Knowledge (AS&K) to merge a comprehensive competency assessment library into Metrics that Matter® analytics technology in order to automate administration and reporting of more than 300 individual assessments. Continued
Republished from ASTD (Online) OD/Leadership News, June 2006
By John Skinner
There are numerous metrics that serve specific purposes. Some of the most critical include performance metrics, operations metrics, financial metrics, and cultural metrics. As the names indicate, performance metrics deal with the actual performance of learning programs relative to a set of criteria, operations metrics describe what is going on in the organization, financial metrics catalogue the investments made, and cultural metrics tell the story about the overall organization.
How to Collect This Data
Once you understand these four categories of metrics, you have several possible methods to collect data, including surveys, interviews, focus groups, and empirical research. Depending on your particular situation, each method has its advantages and disadvantages.
Surveys are easy to scale. Items on the survey should conform to the metrics of interest-and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.
Focus groups are valuable sources of information as they allow you to dig deeper with follow-up questions. The most important thing to remember when using focus groups is structure; you should maintain a list of questions that conform with your metrics and follow it. The drawback to focus groups is that they are time intensive and are not as easy to scale.
Empirical research can serve as a goldmine of information. While this method can be time intensive, the results often are highly credible. Some examples of empirical research include control-group studies or statistically linking training to quantifiable organizational outcomes. You may need to engage an outside consultant to help with an empirical study, but this can be worth it for costly, visible, and/or strategic programs where credibility and precision in the resulting metrics are paramount.
Interpreting the Data
Once you collect the data, you must properly interpret it, aggregate it, and populate the desired metrics.
Aggregation refers to the level you choose to examine the data. For example, if you want to look at a level 1 metric, instructor performance, you can look at the data at the class level: What was the performance for this particular class? You also can look at this from the instructor level: What is this instructor’s performance overall across classes? Having a clear idea of what levels of aggregation may be needed for various metrics before data collection begins is important.
Aggregation can be thought of as filtering the data. You can only filter, or aggregate, based on certain criteria that you collect with the rest of the data. Knowing which metrics need to be aggregated and presented at what levels ahead of time can save you headaches down the road.
Frame of Reference
Even with the best metrics and data, they should not be interpreted in a vacuum. It is essential to establish context for quantitative metrics.
Benchmarking against an external data set from similar learning programs can put metrics in perspective. This information can be crucial to maintaining a human capital edge in a competitive industry. Internal benchmarks also can provide context for metrics.
Finally, the combination of internal and external benchmarks can help set goals. Goals themselves can be an important context for evaluating the performance and effectiveness of the learning organization, as well as a process check when initiating improvement plans.
As you can see, it is crucial to have a strategy in place to effectively leverage learning metrics. If you use the information presented here as a guide to setting your strategy, you will be much better positioned to adapt and scale to meet the needs of your organization and its stakeholders.
Republished from CLO Magazine, April 2006
By John Skinner
A Systematic Approach for Learning Organizations to Assess Needs, Effectiveness, Business Results, ROI and Profit Impact
There are numerous metrics that serve specific purposes. Some of the most critical include performance metrics, operations metrics, financial metrics, and cultural metrics. As the names indicate, performance metrics deal with the actual performance of learning programs relative to a set of criteria, operations metrics describe what is going on in the organization, financial metrics catalogue the investments made, and cultural metrics tell the story about the overall organization.
A constant struggle every learning organization faces is optimizing value for the investment. Optimizing is decision-making based on the past for the future. There must be a process to optimizing value and it must be built into the fabric of the learning organization to ensure it is functioning as designed.
If a process exists to not only quantify value on prior investments but forecast future value the benefits are significant and include:
Surveys are easy to scale. Items on the survey should conform to the metrics of interest-and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.
- Ensures that business results are aligned with business objectives
- Significantly reduces wasted L&D expenses
- Substantially increases productivity
- Simplifies the process of tracking actual business results
- Cost-effective approach to measuring ROI
- Enables you to link learning investments to bottom-line impact
- Data-driven approach to optimizing the impact of your L&D programs
Without a systems and processes in place to optimize value, most organizations are not reaping the full benefit of their Learning and Development (L&D) investments. More importantly, by systematically finding ways to improve impact, organizations can significantly impact earnings. For example, if an organization with $1 billion in labor cost can increase productivity by just 1%, it will increase earnings by $10 million.
Value optimization enables organizations to redirect wasted or poor-performing L&D investments into higher impact programs, which will significantly impact the bottom-line.
Case Study Highlight: The award winning Defense Acquisition University (DAU) led by CLO of the Year Frank Anderson implemented a data-driven approach to analyze value.
Over the course of three years DAU was able to significantly increase output, effectiveness and business impact without any increase in funding. This is a clear example of how an organization can systematically reduce waste and redirect investments into higher impact programs.
Human Capital Contribution Model
A systematic process to valuing learning investments is referred to as the Human Capital Contribution Model or HCCM. The model is composed of 5 core components.
Business Needs Analysis is the up-front planning process used to understand the driving business factors prompting the creation of a learning and development program.
Why is this important? Because the key to a successful learning and development intervention that maximizes impact on the business begins with a formal analysis of business needs.
What’s the process? To ensure the right L&D investment is designed, developed and delivered a needs assessment should be conducted to understand the business objectives. Once identified, a knowledge and skill assessment should be conducted to examine the gap between the current state and desired state.
What are the tools? The primary tools are needs assessments, knowledge and skills assessments, pre and post tests, and competency assessments. Using learning analytics technologies to collect, store, process and report these results, greatly facilitates the process.
What are the results? Better designed learning programs and more clearly defined and measurable business objectives that can be linked to post-learning business results.
A case study is the example of CNA Insurance. The organization was designing a new learning program. A 40 question scenario-based skill assessment was used. The net result was the skill assessment helped in the development and delivery of the training to make it more targeted to the end user.
Performance Analysis is a systematic approach to redirecting poor performing investments into programs that drive on-the-job productivity.
Why is this important? Because research has shown that nearly half (50%) of all training is wasted. In financial terms, for a typical Fortune 500 company that is more than $50 million per year, more importantly, a typical Fortune 500 company could drive another $10 million in earnings if productivity were improved by 1%.
What’s the process? To ensure waste is eliminated and productivity continually improved a systematic review of key indicators is needed to pinpoint the root cause of poor performing investments. Performance analysis focuses on where the investment occurs (Which programs? Which courses? Which vendors? What customers or lines of business?). It also analyzes key attributes of activity and performance data including attendance rates, learning effectiveness, job impact, and value indicators.
What are the tools? Data sources for this analysis include LMS (learning management system) data and evaluation data. Using learning analytics technologies to collect, store, process and report these results greatly facilitates the process.
What are the results? Better information for decision-making. Future allocations of L&D resources to more value-added programs, courses, vendors, and clients or lines of will significantly impact the bottom line.
Case Study Highlight: Nextel Communications, Inc. was conducting sales training. The company hired a third party to assist in delivery of it. Within the first days of the rollout the company was able to quickly review the results of the third party instructors. The results illustrated that certain instructors were poorly performing. The results of the above helped to eliminate waste and improve quality.
Business Result Analysis is a systematic approach to connecting L&D investments to a balanced set of actual results.
Why is this important? Because you can’t manage what you don’t measure. By systematically analyzing the link between learning and business results, one can identify areas for improvement and help drive better results. In addition, by analyzing business results, validation for ongoing investments can be made.
What’s the process? HCCMTM is based on a balanced approach and being as closely aligned to financial statements as possible. After all senior management performance is almost always based on financial results.
The recommended balanced set of business results most closely aligned with financial statements are:
- Revenue
- Profitability
- Productivity
This approach applies to both the public and private sector and is recommended for all major learning programs including corporate universities, leadership programs and business unit programs. When appropriate, it is also recommended that additional business results be tracked such as quality, cycle-time, employee loyalty, customer loyalty and risk mitigation.
What are the tools? Data sources for this analysis include ERP (enterprise resource planning data), financial and cost accounting data, customer satisfaction data, error rate data, CRM (customer relationship management) data, and HRIS (human resource information systems) data. Learning analytics technologies provide L&D managers with templates and wizards to analyze the key business results but is flexible to provide for customized business results analysis and then displays the analysis on interpretive and graphical scorecards and dashboards.
What are the results? A thorough and timely understanding of organizational impact. L&D becomes advisors for improved performance.
Case Study Highlight: PeopleSoft, Inc. (prior to the Oracle merger) needed to tie the investment in training to a core set of business results. The company used both evaluation data where business results were linked to training both at the conclusion of training and months later and also an automated template to link actual business results with training. The outcome was a quantification of results that showed an improvement of 20% in productivity with the training.
ROI Analysis is a process-based approach to determining the financial return on L&D investments given cost considerations.
Why is this important? Because investments should provide more positive financial value back to the organization than the resources consumed in the original investment. It is also a commonly used financial analysis to validate investment decisions.
What’s the process? To ensure L&D management focus on benefit vs. cost leveraging methodologies from experts such as Dr. Jack J. Phillips’ ROI Process helps provide a framework for ROI. The process of collecting, storing, processing, and reporting data that create a monetary cost and benefit from training are inputs to the ROI process and then isolating the benefit to training and adjusting it for bias and conservatism is part of the Guiding Principles of the ROI Process.
What are the tools? Data sources for this analysis include ERP (enterprise resource planning data), financial and cost accounting data, customer satisfaction data, error rate data, CRM (customer relationship management) data, and HRIS (human resource information systems) data. Leverage analytics technologies to wrap automation, templates and wizards around the Phillips ROI Process to make the administrative burden of ROI feasible and practical when there are limitations on resources to apply to ROI analysis.
What are the results? A credible and reliable ROI expressed as a benefit to cost ratio that when compared by learning delivery, program, vendor, client or line of business will help the L&D manager make better decisions from a dual dimensional perspective (benefit and cost). It will also validate the investment (or invalidate it in the case of a negative ROI).
Case Study Highlight: New Horizons Computer Learning Centers, Inc. was assisting a client in technology training. Management was very interested in the financial ROI.
The key focus was on estimating the change in performance before versus after the training and isolating it to training and then adjusting it for confidence and conservatism. The cost of the training was also conservative by design.
The outcome was an ROI that was positive and produced future economic value as seen through a 3.43 payback period.
Profit Impact Analysis is a data driven planning and reporting process that helps to optimize the impact of L&D investments and to connect learning to financial statements.
Why is this important? Because L&D expense is part of the calculation, these profit measures are arguably the most meaningful ways to analyze past and future impact of learning on earnings.
What’s the process? If learning intervention is effective, it will ultimately impact profit in a positive way. The financial goal of a corporate university is to increase the profit contribution of its people. That profit contribution can best be measured by taking revenue less labor costs and L&D expense, which is called The Human Capital Contribution Profit. A corporate university will be most successful when the Human Capital Contribution Profit is growing faster then the revenue, which is measured by calculating the Human Capital Contribution Margin (Human Capital Contribution Profit divided by Revenue).
What are the tools? Data sources for this analysis include financial and accounting systems. Learning analytics technologies can provide templates to guide a manager through the inputs and structured analysis to trend and track the actual to projected results.
What are the results? A sensitivity analysis tool that can help plan investments better and optimize L&D impact. It also enables learning professionals to discuss financial impact with business executives.
Case Study Highlight: KnowledgeAdvisors, Inc. a learning analytics technology company reviewed the profit impact from a major investment in a new technology platform and analyzed the training effect on the investment. The company took a financial approach to analyze this important learning and development investment.
Projections were made for the 2nd and 3rd years after the training as management set out to prove that although the training investment would be significant it would pay off in the form of future earnings that directly impact the bottom line. The second year projection using a profit impact analysis approach was a 6% improvement in profit margin. This was because the new technology platform was projected to be brought online quicker and managed more effectively as a result of the training (the human capital contribution effect) and ultimately help accelerate future sales by 5%. The 3rd year was projected to increase profitability by 3% with a 3% increase in revenue due to the training investment.
Conclusion
Optimizing value in learning and development investments is not easy. However, it is not impossible. The right systematic process, technology and templates can facilitate the analysis.
In today’s world of limited money, time and personnel we face an era of accountability for all that we do. If we are to be the fiduciary of millions of dollars for learning and development, management should expect nothing less than 1) a reasonable analysis on the business needs that drive the investment, 2) the manner in which we will optimize the performance of the investment so as to lower our risk of wasted training, 3) the linkage of the investment to key business results like revenue, profitability and productivity, 4) a reasonable ROI calculation based on the isolated and adjusted benefit from training, and 5) the projected impact on future earnings.
Based on observation in other lines of business such as sales, marketing, production planning, warranty and claims etc. these analysis have been done for years for equal strategic, visible or costly investments. Learning and development organizations should leverage the best practices of other lines of business and derive roughly reasonable ways to optimize value. Business is not about precision and perfection it’s about how to provide reasonable information in a timely manner to make better decisions. Keeping this in mind will go a long way in demonstrating accountability while illustrating how the investments will generate the most for the stakeholders.
