Posts Tagged ‘learning measurment technology’
Republished from: Measurement, Evaluation and ROI News, March/April 2006
By Jeffrey Berk
In today’s world of learning analytics it is easy to get caught up in a whirlwind of ROI, linkage to business results and bottom line impact terminologies. However, what about the basics of evaluation? How can 21st Century technologies make this simple yet powerful process more practical and scaleable within a learning and development (L&D) organization where they just need to see the results of a Level 1 Reaction survey or a Level 2 test? This article will take us back to the basics of evaluation by exploring the core components of the process of evaluation and offer tips and examples of how to make it easier where there are limited financial, physical and human resources to accomplish the basic process of evaluation.
Data collection:
There are a host of inexpensive web-based data collection tools. These tools are easy to use and allow you to collect data from learners, instructors, managers, and any stakeholder at any point in time. Leveraging the internet to collect data saves costs of paper processing. It is worth investigating and leveraging where practical and feasible. Response rates are not dramatically different (65% online / email vs. 85% paper based on research). Even if paper is necessary, scanning technologies can easily scan quantitative and qualitative data into databases for centralized data storage and processing.
Technologies specifically designed for learning evaluation will provide standard evaluation tools to take advantage of benchmarking. They will also have multiple ways to get data into the storage component (discussed below) such as online, email, data entry, and data import.
Data Storage When it comes to data storage, I don’t mean that metal file cabinet in the basement closet of your office building that so many end of class evaluations go to as their final resting home. Simple tools like spreadsheet applications can be inexpensive and powerful technologies where data collected online can be dumped for efficient, centralized data processing. More sophisticated solutions include using local relational databases such as Access that is more powerful than Excel or using enterprise relational databases such as SQL or Oracle that can span the organization easier. Data storage in technology tools is central, secure, and makes processing easier than that basement file cabinet.
Technologies specifically designed for learning evaluation will have centralized data storage to warehouse your evaluation data, setting you up for success in data processing and reporting.
Data Processing
Data processing is probably the most time intensive piece in all of learning measurement. The key here is to have flexible tools that allow for the building of queries that can then be automated or standardized. OLAP tools such as Cognos or Microsoft Analysis Services are very powerful for querying large amounts of data. However, a strong word of caution. Do not throw technology into the hands of the functional users untrained on learning measurement. Allowing a line of business person to write their own OLAP query that compares learning data to business data can be counter productive if not dangerous if they do not know what they are doing.
Technologies specifically designed for learning evaluation will have a very user friendly front end to allow a learning consultant to query their Level 1 evaluation data by attributes of the training (instructor, course, location, date, delivery) and by attribute of the trainee (business unit, client, job function etc.). It is just a matter of making easy selections in a wizard-like tool and the output is exactly what you need.
Data Reporting
In most organizations there is a need for 4 primary types of reports. Your measurement solution needs to be able to provide reports for each. These reports include tactical, aggregate, executive, and value analysis.
Let’s start with tactical reports. These are for the staff personnel in the training department that need to manage the learning investments on a day to day basis. Examples of tactical reports include class evaluation summaries, respondent qualitative verbatim, and the actual evaluations themselves. These are used to spot problems before they can get any bigger.
Next are the aggregate reports. Middle managers in a learning group use these reports. They aggregate tactical data for monitoring and quality control. For example the person responsible for managing the sales curricula needs to quickly view all the course titles and determine which are most effective. The person managing the eLearning content needs to easily see how effective it is compared to traditional instructor-led training. These managers need to see tactical data rolled up and they need to be able to filter it down too. Reports for these folks should do this.
Then executives in the learning group need reports. They need to see comparisons internally and externally. For example how did employees in each line of business that fund the training see the training impact their jobs? How did the key indicators of training compare to external benchmarks? And what exceptions occur on a day to day basis? Executives need summary, exception-based data that is comparable internally and externally.
Finally there is value analysis. This is packaging it all together. It is your balanced scorecard that you can present to management when the budget is up for renewal or you need to fund a new program. It’s the measures you have an appointment with CEO to review on a quarterly basis cut by eLearning vs. ILT so the CEO can see how that eLearning investment is doing. Value analysis sums it all up. It presents all the measures in an easy-to-generate and easy-to-interpret score-card. Value analysis allows a chief learning officer to sit down with a C level person or a line of business manager to cover how training impacted the job, how human capital was improved through training, and yes, the financial return on training in hard and soft dollars. Metrics such as benefit to cost ratio, payback period and ROI percentage are in the value analysis.
Technologies specifically designed for learning evaluation will have canned reports that output to these types of analysis. They will allow for easy configuration of those reports to ensure the data in the report is relevant to the need. Finally they will have multiple output formats such as PDF, web, printer, email, and excel.
Conclusion
Basic evaluation does not need to take considerable money, time and people. In today’s world of technology, even a small learning department can leverage technologies specifically designed for learning evaluation to process thousands of evaluations for a cost that is likely to not even be a rounding error in their learning and development budget.
It is well worth the time to explore a learning analytics solution to outsource the basic administration of evaluation. This will allow the learning organization to focus on value and continuous improvement
About the Author
Jeffrey Berk is the Vice President of Products and Strategy for KnowledgeAdvisors, a learning analytics technology company. KnowledgeAdvisors helps organizations improve the efficiency of the basic evaluation process through easy-to-use technology tools specifically designed for learning evaluation.
Metrics that Matter® Learning Analytics at a Major Financial Services Company
Company Overview: The organization is in the financial services industry and is one of the top 25 largest banks in the United States. The company employs over 15,000 people and has over 1.5 million customers. Continued
Pharmaceutical Company Uses Metrics that Matter® to Validate the Value of Its Leadership Program
Company Overview: The pharmaceutical firm recognized that cultivating leaders was important to the success of its organization. As a result, the company established a leadership program to build organizational bench strength and groom future leaders. Continued
Ninth House Measures Success with Metrics that Matter®
Company Overview: Ninth House was founded in 1996 and specializes in a unique blended learning approach to leadership development. Continued
Hospitality Company Uses Metrics that Matter® to Monitor and Track Learning Initiatives
Company Overview: The organization is a leading hospitality company with over 2,000 properties in over 60 countries. Continued
Leveraging Learning Analytics Technology at Caterpillar
Company Overview: Caterpillar, Inc. is a manufacturer of earth moving equipment. Caterpillar employs over 80,000 people around the world. They have 277 locations and nearly 100,000 dealer employees. Continued
