Posts Tagged ‘Learning Analytics’

Republished from  ASTD (Online) OD/Leadership News, June 2006

By John Skinner

There are numerous metrics that serve specific purposes. Some of the most critical include performance metrics, operations metrics, financial metrics, and cultural metrics. As the names indicate, performance metrics deal with the actual performance of learning programs relative to a set of criteria, operations metrics describe what is going on in the organization, financial metrics catalogue the investments made, and cultural metrics tell the story about the overall organization.

How to Collect This Data

Once you understand these four categories of metrics, you have several possible methods to collect data, including surveys, interviews, focus groups, and empirical research. Depending on your particular situation, each method has its advantages and disadvantages.

Surveys are easy to scale. Items on the survey should conform to the metrics of interest-and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.

Focus groups are valuable sources of information as they allow you to dig deeper with follow-up questions. The most important thing to remember when using focus groups is structure; you should maintain a list of questions that conform with your metrics and follow it. The drawback to focus groups is that they are time intensive and are not as easy to scale.

Empirical research can serve as a goldmine of information. While this method can be time intensive, the results often are highly credible. Some examples of empirical research include control-group studies or statistically linking training to quantifiable organizational outcomes. You may need to engage an outside consultant to help with an empirical study, but this can be worth it for costly, visible, and/or strategic programs where credibility and precision in the resulting metrics are paramount.

Interpreting the Data

Once you collect the data, you must properly interpret it, aggregate it, and populate the desired metrics.

Aggregation refers to the level you choose to examine the data. For example, if you want to look at a level 1 metric, instructor performance, you can look at the data at the class level: What was the performance for this particular class? You also can look at this from the instructor level: What is this instructor’s performance overall across classes? Having a clear idea of what levels of aggregation may be needed for various metrics before data collection begins is important.

Aggregation can be thought of as filtering the data. You can only filter, or aggregate, based on certain criteria that you collect with the rest of the data. Knowing which metrics need to be aggregated and presented at what levels ahead of time can save you headaches down the road.

Frame of Reference

Even with the best metrics and data, they should not be interpreted in a vacuum. It is essential to establish context for quantitative metrics.

Benchmarking against an external data set from similar learning programs can put metrics in perspective. This information can be crucial to maintaining a human capital edge in a competitive industry. Internal benchmarks also can provide context for metrics.

Finally, the combination of internal and external benchmarks can help set goals. Goals themselves can be an important context for evaluating the performance and effectiveness of the learning organization, as well as a process check when initiating improvement plans.

As you can see, it is crucial to have a strategy in place to effectively leverage learning metrics. If you use the information presented here as a guide to setting your strategy, you will be much better positioned to adapt and scale to meet the needs of your organization and its stakeholders.

Continue Reading

Republished from CLO Magazine, July 2005

By Jeffrey Berk & Scott Magee, KnowledgeAdvisors

Learning Analytics technology is an enabling tool that can significantly assist learning organizations in understanding how to better train and develop employee’s partners and customers. The building blocks of learning analytics technologies include the way analytic data is 1) collected, 2) stored, 3) processed, and 4) reported. Mapping out the points of automation for each of these is helpful when designing a learning analytics technology.

Surveys are easy to scale. Items on the survey should conform to the metrics of interest-and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.

Data Collection

Data collection is how the organization obtains its original data for storage, processing and reporting in the analytics tool. A major technological consideration in data collection is to leverage online and email data collection devices. Yes, response rates may drop but timeliness of processing and reporting will go up and costs will be saved. Eaton Corporation’s Eaton University is a great example of technology for data collection. According to Joyce Gilman, Operations Manager of Eaton Corporation, “All data for the University is collected via email for end of class, follow-up and manager evaluations. This saves us significant time and cost while continuing to provide Eaton University a significant amount of data to analyze for continuous improvement opportunities.”

But what if your organization cannot collect the data electronically? Either the culture or the physical infrastructure makes this an impossibility. Scanning technologies can be leveraged to import raw data into analytics technologies. “Although we collect our learning evaluations in a paper-based manner, we have systematized the processing of the data via scanning technologies. The raw data is batch-imported into an analytics tool that processes and reports the data. As a result, our learning organization focuses on the results and quality of the training, not compiling numbers,” commented Robin Killeen, Project Manager for Discover Card.

When collecting data it is important to be mindful of data security as well as data privacy and protection. Physical security controls should be in place to prevent unauthorized access. Network security is equally important, ensuring that firewalls are in place to prevent unauthorized access. Data privacy and protection means that respondents have the option to submit data anonymously and they have knowledge of how the data will be used. Conforming to rules such as the Department of Commerce Safe Harbor legislation is a way to provide comfort to users that the data collection site has been certified by a regulatory body as safe in the protection of data privacy.

If the source of collected data is in question it is not worth the pain of readying it for analysis or in corrupting an existing database with bad data. Collecting data used in analysis that has credibility and integrity is important. A best practice company is Hibernia Bank. According to Robert Baer, Design Team Leader, Hibernia’s learning organization knows exactly the format and source of real business results produced by various systems and technologies. “We use this data to accurately and credibly link the results to training programs. This makes the analysis well received by our senior management.”

A final technological consideration is the configurations and settings. Establish minimum configurations for browser and operating systems that are commonly used within your organization and business in general. For example only supporting Windows XP when the majority of your workforce is on Windows 2000 is not a flexible minimum supported option to rollout to your workforce.

Data Storage

Data storage describes the environment the organization uses to maintain its learning analytics data. The data structure and use becomes a significant technological consideration here. Really knowing how the data will be stored and what its use will be is critical. GTECH, a leading provider of gaming and technology solutions worldwide, has a detailed outline of how its training courses fit into hierarchical curricula and programs. Margaret Lamb, a GTECH Instructional Designer, said, “Knowing that our analytics technologies can support how we organize our curricula in its data structure is very important. This means that we can measure segments of our course offerings, as well as the total program. This strategy really helps us understand and manage our learning programs.”

Centrally storing all of your collected data in a single data warehouse is critical to the processing and reporting elements of analytics. Large IT hardware and software vendors often train high volumes of end customers through third-party partners around the world. Many leverage a third-party learning analytics tool that acts as a clearinghouse to gather the data from multiple decentralized locations and then centrally stores it in a single database. The central storage of the data then allows these vendors to access all of this data easily and in a timely manner to better manage quality, performance and value aspects of their training.

Analytics tools rely on large volumes of data to allow users to slice and dice in multiple ways. Each data point must be stored in its raw format in the database. This means that basic tools like a spreadsheet application may have size limitations on data storage. It makes sense to therefore store the data in more powerful tools such as OLAP systems that are great are powering through large sets of data. New Horizons Computer Training Centers is a great example. This professional learning organization has collected over 1.5 million evaluations in the past few years. When responding to a recent RFP, they were able to quickly query the OLAP database and find out that the prospective customer had received training from New Horizons over 100 times and in 30 locations around the world. This made for a more impactful response to the RFP and the data was gathered in minutes.

Retention of data coincides with storing it. Over time, large amounts of training data can consume even the most powerful databases and slow them down. Establish a data retention policy early in the analytics process and communicate it to users of the analytics technology.

Finally, technology comes with the risk that data can be lost or corrupted. In order to mitigate this risk, establishing backup procedures is one technology consideration not to be overlooked. An hourly or more frequent transactional backup of data is important. Nightly full backups are recommended. Ensuring that the backups cannot be overwritten is a further precaution. Finally, taking the backup files to offsite locations is a physical security procedure.

Data Processing

Data processing describes the actions taken on the stored data to transform them into business intelligence ready for analysis. A primary element of processing is to address data formatting issues. Understand the native format and origin of data before including it in an analytics engine. This will help in determining the right option for exporting and manipulating the data. For example, data in a Relational Database Management System (RDBMS) is easy to de-normalize and analyze in other RDBMS’. But data being imported for processing from a static Excel spreadsheet may be more challenging to upload and analyze in an RDBMS. The car maker Audi worked closely with technologists to appropriately format data derived from these systems to provide the analytics technology a common file feed. According to Annette Eagle-Dull of the formerly of the Audi Academy, “This file feed is now being customized for import into an analytics tool. Once done Audi will be able to analyze their data in a single platform with multiple filter options and make the analytics tool accessible to their instructors and designers for quality analysis.”

OLAP capabilities were discussed a little bit in the data storage considerations but are equally important to discuss in processing considerations. Using sophisticated Online Analytical Processing (OLAP) tools to process data increases the power of analysis. There are a myriad of OLAP tools on the market that range in price and functionality. The key element here is to build the right structures within the OLAP tool to allow a functional user to easily manipulate the data self-sufficiently. This often requires front-end interfaces to be built on top of these tools eliminating the need for IT departments to use tools like this to write reports each time a user needs them. Nonetheless the power of OLAP tools is important in a user ability to ‘slice and dice’ the data by key learning and organizational elements. Examples of filters in the learning space include:

Instructor
Course
Class
Curricula
Program
Location
Learning Delivery
Start Date/End Date of Training
Type of Evaluation
Question/Question Category
Business Unit
Customer/Client

A leading professional services firm’s training operation views its learning metric data through an internet-based OLAP tool that has the capability to compare performance across multiple attributes of its learning operation and its business operation. According to a director the firm’s global learning, queries can be run by filtering on course, instructor, and delivery components of training but also by line of service, sub group and industry type which help the learning organization understand the impact training has not only on training attributes but also on key components of the business.

Netxel Communications, Inc. leverages the technological consideration of processing consistency in its analytics tools. Per Danny Brown, Manager Evaluation and Metrics, “All learning metric data is processed from a single point of origin. This is made possible by the analytics technology’s preconfigured queries that reach back to the stored raw data to perform queries not calculating from previously calculated data which can skew results.” Brown articulates the importance of relying on analytics tools that conform to processing consistencies. “You need to be comfortable that the technology will always run the same query in the same way no matter what the data set or time frame is going to be,” said Brown. Ensuring the queries are preconfigured across the technical architecture of the analytics tool to be consistent is important to system integrity.

Technologists will need to understand the syntax languages of the various analytics tools that are being utilized to store and process the data (Access, SQL, Excel, SPSS, SAS, etc). Resources will be needed that are knowledgeable regarding ETL data (Extract, Transform and Load). ETL is a data integration function that involves extracting data from outside sources, transforming it to fit business needs, and ultimately loading it into a data warehouse. This is where data is transformed from one native language to another so that it can be processed and analyzed in an analytics tool. If technologists are not well versed on ETL and syntax there is a higher risk of misinterpretation and use of data types in the analytics tools.

There are various feeder systems to a learning analytics tool. HRIS and LMS systems are the two primary feeder systems that supply data to a learning analytics tool. Leveraging a standardized XML schema can allow the extraction of data in one system and import/storage of that data into another in an automated and friendly manner. For example, Caterpillar Inc.’s learning organization used a standard schema to export learner demographics and class attributes housed in the LMS to their analytics technology tool. According to Veronica Schmeilski, the Caterpillar project manager on the integration, “The IT organization was provided a standard XML schema by the analytics vendor to extract details from the LMS (registration, completion, and demographic data). Nightly this data is sent via FTP to the analytics vendor where it is automatically picked up and imported into the analytics tool that then initiates evaluation collection, storage, processing and reporting protocols.” Overtime the schema changes have occurred as the business and learning evolves to make it further customized to Caterpillar’s needs.

Data Reporting

Data reporting describes the formal presentation of the results of a processed request. A major technological consideration is ease of use. This may seem functional rather than technical. But, ensuring the functional users can see results in a manner that is easy to interpret is important. Using tables and charts with appropriate language surrounding the results mitigates risk of misuse. Investing in charting software or other packages that can appropriately display results is important. Defense Acquisition University (DAU) uses easy to read charts (red/yellow/green – stoplight charts) that any report user can quickly understand to identify strengths and areas of opportunity. Mark Whiteside, DAU Director of Performance and Resource Management ensures that easy-to-interpret reports can be run across multiple areas of DAU training such as by instructor, location, course, and delivery. Per Whiteside, “We want to maintain our position as a world-class learning organization. Ensuring we have data that can be quickly interpreted on the key elements of our quality process is critical. The format of our stoplight-type reports makes them easy to evaluate performance in an efficient and effective manner.”

Quality in the learning analytics technology ensures credibility in the technology. Changes when needed to the reporting environment should first be done in a safe test site to ensure the change is appropriate then migrated to the production site, the official reporting site. This is done to ensure functionality and security is not adversely affected. Also regression test all new code releases prior to implementation into the reporting environment. This means test prior functionality even if not presumed to be affected by new functionality. This minimizes negative impacts new functionality has on existing functionality.

Aligned with the quality process is communication of maintenance windows and escalation paths for customer support. If the reporting application needs planned maintenance communicate the scheduled downtime to users of the application in advance and make the change takes place during the least obtrusive time. Support the reporting application with technical help. It is advisable to establish appropriate escalation paths that define the levels of support. For example, Tier 1 support could be basic technical glitches that are not impacting all users and to which there are workarounds. A more significant escalation could be an application outage affecting all users. The owners of the learning analytics tool should establish a supporting process so users of the learning analytics technology have a means to report and receive feedback and have the appropriate support points of reference depending on the technical problem.

Conclusion

Learning analytics has emerged as a powerful business decision-making tool for learning operations managers. If built with the correct technological considerations these tools can offer timely and accurate business intelligence to help provide answers in terms of learning activity and performance.

Grant Thornton’s corporate university understands the power of metrics and the technology considerations to make it possible. Bob Dean, Chief Learning Officer for GTU (Grant Thornton University) established the cornerstones of linking training to business results and showing value as part of his learning strategy. Dean, a visionary in this field, leverages virtually all of the considerations discussed above. “At GTU we want to focus on proving the value not processing the data. So we collect all of our metrics online. We have integrated our analytics system with our learning management system so we centrally store our analytics data and let the analytics tool process it. We then leverage the data when making courseware decisions and in showcasing the power of training to our management.”

The key with any analytics technology is ensure it can automate as much of the data collection, storage, processing and reporting as possible. Doing the automation in a way that makes learning analytics practical, reasonable, cost effective, and repeatable is the benchmark for successful implementations of this technology.

Continue Reading

Republished from HR.com, March 2004

By Jeffrey Berk

In the past 12 months a tidal wave of information has been generated by the learning industry on the need and power of learning analytics. Learning Analytics is the set of activities a learning organization does that helps it understand how to better train and develop employees and customers.

A survey was prepared and data was collected. Over 100 training departments responded from a variety of industries. Now the results are in.

The results are organized around the key constructs we used in our survey. Looking at technology, process, stakeholders, and strategy.

Each week Hr.com will provide a look into the current trends and future state of learning analytics for each of these key constructs.

Technology

This article takes an in depth look at how technology can be leveraged make the learning analytics process more automated and streamlined.

Current Practices from the research:
The summary points below articulate the way organizations leverage technology in their learning analytics process today.

• Most organizations still leverage paper as the primary data collection technique when gathering metrics on learning investments.
  About one quarter of the organizations do not store learning measurement data in a centralized database.
• Two thirds of organizations use spreadsheet applications as their database of choice when storing metrics on learning measurement data.
• Just over one third of organizations have the capability to compare learning measurement data by learning delivery method
• Little or no automation is leveraged by respondent organizations to filter and query the metrics.
• Just over half of the respondents have the capability to import data from a learning management system into their evaluation and measurement systems.

Best Practice Improvement Opportunities:

• The following points highlight what best practice organizations are doing to optimize how they leverage technology for their learning analytics process.
• Leverage technology throughout all elements of the measurement process. There are four primary elements of measurement: data collection, data storage, data processing and data reporting. Technology should make each of these more efficient. The result is minimized administration time preparing metrics and maximized time doing something with them.
• Use a centralized database to store and query your data. Powerful tools (i.e. OLAP) tools exist to query large amounts of data. But, be careful. Technology is a mere enabler. Throwing a powerful query engine at a novice user can be dangerous. You’ll need to build the right front end interfaces and back end reporting around your query engine to maximize its usage.
• Technology should be leveraged to ‘tag’ all the data elements you want to report.The most practical ways to manage a learning organization are to slice your data by learning provider, location, business unit, class, course, curricula, learning delivery, and instructor.
• Learning analytics systems should ‘talk’ to other systems such as an LMS. Using code like XML can link the systems and minimize redundant tasks and mitigate risk of error.

Conclusion:
Technology is extremely important to optimizing your analytics solution. A small investment in technology can pay enormous dividends short and long term. Technology exists today that is very affordable (usually less than 2% of your training budget) to comprehensively streamline your measurement solution.

Process

This article takes an in depth look at the key inputs, activities, and outputs that comprise the learning analytics process.

Current Practices from the research:
The summary points below articulate the elements of today’s learning analytics processes.

• Just under two-thirds of responding organizations measure 76 to 100% of their training in a formal manner.
• Only one third of responding organizations have a standard set of key performance indicators that they regularly measure and monitor on their training investments.
• Most all organizations use an end of class ‘smile sheet’ instrument to evaluate training but less than 30% use on the job data collection instruments that go to the participant of the training and their direct supervisor.
• Over half of responding organizations have budgeted 5% or less for learning measurement. Nearly one quarter have no budget at all for learning measurement.
• Over one third of responding organizations have no full time equivalent resources to focus on learning measurement within their organizations. Most have one or two.

Over 80% of a respondent organizations resources for measurement is consumed in administrative aspects of measurement leaving less than 20% to actually improve training from the metrics or use the metrics to show value to stakeholders.

Best Practice Improvement Opportunities:
The following points highlight what best practice organizations are doing to optimize their learning analytics processes.

• Leverage a standard set of key performance indicators linked to industry accepted learning measurement models (i.e. Kirkpatrick Learning Levels) that are practical and scaleable indicators and predictors. Collect this data for each class and then aggregate it, and slice and dice it on a formal performance scorecard.
• Leverage the use of impact data instruments not solely smile sheets. Impact instruments look at job impact, business results and ROI when the participant is back on the job and in the best position to provide feedback on these more critical elements of measurement. Sending a similar instrument to the managers also reinforce your measurements.
• If you use technology to automate the collection, storage, processing and reporting of the resulting data then the process is optimized.

Dissect each step of the measurement model. Look at how you can leverage automation and technology and streamline steps for collecting data, storing data, processing data, and reporting data. Best practice companies that do this actually spend 20% of their time on admin activities and 80% on using the metrics to improve training and demonstrating value to stakeholders.

Conclusion:
Appropriate processes that identify measurement strategic objectives and then measure against those using technology to do the heavy lifting is critical to success. Equally critical is building processes wrapped around not only technology but industry accepted methodology for learning measurement. This is important because it helps with change management and credibility concerns.

Stakeholders

This article takes an in depth look at the critical consumers/customers/users of learning analytics reports and data.

Current Practices from the research:
The summary points below articulate the elements of today’s learning analytics stakeholders.

• Performance analysis (showing impact, results and ROI) is the single item where the largest gap exists whereby learning organizations are poor performers in this area but stakeholders feel it is extremely important.
• Over two thirds of communications to stakeholders are periodic meetings and conversations or a management report.
• Nearly one third of communications to stakeholders are done on an as requested basis which is often a reactive communication style.
• The largest consumers of learning analytic data are training staff and management.
• The top reason responding organizations cited for measuring their training was to showcase the value to the organization.

Best Practice Improvement Opportunities:
The following points highlight what best practice organizations are doing to optimize their communications with stakeholders.

• Find practical, scaleable, and replicable ways to do performance analysis (impact, results and ROI metrics) so that stakeholders get timely information that is significantly more salient to the reasons why they budgeted for the training.
• Leverage proactive communications to stakeholders such as automated ‘push reporting’ and access to online measurement reporting systems to allow stakeholders with metric data in a self-service model.
• Showing measurement data to stakeholders using the right metrics (impact, results, and ROI) is a great way to expand the use of the metric data beyond the walls of the learning department.
• Because the most significant factor in going through the process of learning measurement is showing value to stakeholders, ensure what you show to illustrate value is important to the stakeholder. Focus on linking training to their business objectives, illustrating the behavior changes made on the job, and quantifying the financial ROI on the investment.

Conclusion:
This study and other independent research shows that a major priority of learning organizations is to prove the value of training. In a world where training must compete with marketing, sales, and other department dollars, providing the value of how those dollars will be used is essential. Deploy practical approaches to gathering and presenting this data to the stakeholder to convince them that the training did have value and that future investments are well worth it.

Conclusion:
Appropriate processes that identify measurement strategic objectives and then measure against those using technology to do the heavy lifting is critical to success. Equally critical is building processes wrapped around not only technology but industry accepted methodology for learning measurement. This is important because it helps with change management and credibility concerns.

Strategy

This article takes an in depth look at how technology can be leveraged make the learning analytics process more automated and streamlined.

Current Practices from the research:
The summary points below articulate the way organizations derive learning analytics strategy today.

Only 18% of responding organizations measure the ROI on their training investments
Only 30% benchmark training investments externally by industry, job function etc.
Only 14% of responding organizations have analytics tools that are scaleable, and replicable to use for day-to-day measurement.
Nearly 80% of responding organizations feel that reasonable quantitative and qualitative data is required by stakeholders as opposed to the need to get highly precise metrics that may take more time and money to generate.
Nearly two thirds of respondent organizations have very little or no additional funding to generate highly precise metrics as opposed to reasonable indicators.

Best Practice Improvement Opportunities:
The following points highlight what best practice organizations are doing to optimize their learning analytics strategies.

• Deploy a measurement strategy that is based on industry accepted measurement methodology but is practical to implement within the organization leveraging technology to increase practicality. In this way ROI, impact and result driven data can be captured and reported more efficiently and effectively.
• Training investments should be benchmarked internally and externally so the organization can use it as a constructive tool to set goals and objectives, continuously improve and motivate by example.
• Take advantage of reasonable indicators and predictors versus feeling the obligation to deliver highly precise data that may consume large amounts of resources and not be timely enough for decision-making purposes. The reasonable indicators, however, should still link to industry accepted methodology for measurement but do so in a practical manner.
• Using reasonable indicators coupled with technology will accomplish your learning analytics strategy at a cost that is likely to be less than 2% of the training budget. Independent studies have shown that organizations should be investing 3-5% in measurement of learning.

Conclusion:
Strategy creation begins with the end in mind. How does the learning measurement strategy tie to the business objectives? How will you ensure that learning programs do tie to business objectives in practice? These are key questions that need to be addressed when creating a measurement strategy. Finally, strategy needs to be attainable from an execution perspective. If your measurement processes and technologies cannot accomplish the measurement strategy there is a disconnect that needs to be addressed.

Continue Reading

Republished from: The eLearning Developers Journal, November 3, 2003

By Jeffrey Berk

A few weeks ago I was talking to a colleague, Toni Hodges, who is an independent consultant that helps organizations with learning measurement. She and I were discussing the difference between what I referred to as ‘activity measures’ vs. ‘performance measures’ and she quickly summarized my terms into ‘how well you train vs. how much you train’ and the terminology stuck with me. So, this article explores this very issue: focus training measurement on how well instead of how much. Quality and impact versus quantity and butts in seats is what matters in today’s world. Astute users of metrics will be the first to understand the difference so those who produce the metrics should be making adjustments in their measurement plans to ensure the right metrics are being gathered.

Activity measures vs. Performance measures

It seems sensible to start with the difference between activity measures and performance measures. Activity measures tell you how much you have trained. Examples of activity measures include average number of students per class or facility utilization. A more comprehensive list of activity measures is included in table one below.

Performance measures tell you how well you have trained. Examples of performance measures include time to job impact, change in strategic results isolated to the training, or instructor performance. A comprehensive list of performance measures is included in table two below.

Activity measures tend to be easier to collect than performance measures. Most learning management systems can gather such data although it is debatable how well they perform at reporting the data for useful analysis. Activity measures are primarily data components of the training registration process. Because training needs to be coordinated and schedules the data is there to data mine. Most organizations do a fairly decent job of reporting some activity measures that are easy to obtain such as number of classes ran, or number of people trained.

Performance measures are more challenging to obtain. Some feel that certain performance measures are nearly impossible to obtain. Satisfaction-type measures such as instructor performance or courseware quality can be obtained through end of class evaluations often referred to as ‘smile sheets.’ Knowledge transfer can be obtained through pre and posttest scores but the testing exercise can be a significant drain on a company’s resources often not making it a practical tool to use across the board for all training. Still yet, training impact on the job and on specific business results baffles many with not only how to do it but how to do it without spending more on the measurement exercise than the cost of the training itself. Finally, ROI, often thought as the holy grail of all training measures seems so distant to many training groups it is nothing more than a dream.

Nonetheless, performance measures are critical. If you don’t know which programs had the greatest impact on the job and the company’s business objectives your measurement system has some significant chinks in its armor.

Call to Action: Performance Metrics Needed

The new millennium is a completely different workplace than even a short time ago. In the late nineties the economy was booming, the money was flowing, and training had a blank checkbook to experiment with all kinds of new programs. It was not all bad, eLearning spawned as a result of this and continues to grow year after year as a valuable learning delivery method.

However, the bubble burst. The dot com hey day is over. The economy dried up and senior management went with a back to basics way of doing business. At the same time massive layoffs occurred and departments were asked to do more with less. Finally, all budgets have been under a microscope of scrutiny. The training department is not immune to this treatment. As such, the training department can no longer continue to justify its existence with solely activity measures.

Think of this, the CEO of a company is trying to control spending and costs. She asks the chief learning officer for a training budget. That budget is prepared and put forth. The CEO wants some evidence to justify not cutting the budget this year. Reasonable business decisions dictate that tools that help the business perform better and improve the bottom line are good investments. The chief learning officer tells the CEO that they trained 80% of the workforce in the prior year, that nearly 300 classes were taught and that anecdotal evidence suggested people really liked the training. As the CEO, this is not a reason to renew a significant budget line item.

Why was the CEO not compelled by the metrics the chief learning officer gathered? They are quantitative and you can make pretty graphs out of them and do all kinds of statistics on them. The reason is because the CEO was not told that the training supported the business objectives of the company. Did the training help increase revenues? Did it help decrease costs? What kind of impact did it have on the average employees job performance relative to the salary paid to the employee? None of that data was collected, measured and presented.

The bottom line is the chief learning officer’s budget is on the table to be cut. If the right performance metrics had been gathered all along, the chief learning officer would be sitting at the table with the CEO talking about the next training program that is predicted to increase customer satisfaction or reduce cycle time.

Senior executives care about the tools that help make a difference on the job and on the business results. Articulating that your training department has just reached a new all time high for the number of times Typing 101 was taught is not a compelling metric to a senior business leader looking to see how each budget line item adds value to their business.

Hopefully the example above sticks. Hopefully it creates a compelling story to gather performance metrics. Although more challenging, it can be done. But how? Remember you’re living in a world where your budget to conduct your core business, training is at risk. You’re not about to allocate significant resources that are already at risk to an expensive measurement solution that may only measure 5% of your programs. What do you do?

Choose Offense Rather than Defense:

Corporate universities may have found themselves in the situation I just described above. The worst case you can be in is when you are on the defensive. This is the scenario where you got your budget cut because you did not showcase the value of training as a strategic tool to improving job performance. The marketing group retained their budget. They did analysis showing how advertising improved sales. The purchasing group showed how their vendor management tactics saved costs. But the training group failed to show how they improved the human capital they trained.

So now the search is on. Determined not to let this happen, to not sit idle, a best practice-training department sees what is happening around it. They know other functions within the organization are proving value. Those other departments are figuring out how to practically showcase their value through scaleable, replicable, and practical measurement solutions. They are not only doing this to justify their existence and mitigate budgets at risk, but to use such measures as tools to continuously improve and proactively showcase value to senior leaders.

The best practice chief learning officer wants to be on the offensive. This person knows it is not prudent to keep providing the same activity metrics from years past that don’t work in today’s business world. They also know that if senior management forces the issue they’ll be on the defensive, which is not a good position to be in.

So, the best practice chief learning officer will search for the solutions. Key ingredients of the solution include taxonomy, process, methodology and technology. All of these ingredients are absolutely critical to put in place a cost effective yet compelling learning measurement solution. But, it is either that or sleepless nights worrying about budget risks. What would you do?

The Search is On: Finding the Right Model

I have had the fortunate or unfortunate experience of seeing managers in the learning department devise measurement strategies. I have authored a few myself as well. The first step in the process is to research learning measurement taxonomies and models. After all, why start with the blank sheet of paper when clearly others have solved this problem before you were faced with it. In doing your research you are likely to run into many models by PhD’s and other really smart people who claim their approach is the right one. However, one model that has stood the test of time is the Learning Levels model by Dr. Donald Kirkpatrick. It started as a series of articles created in the 1950′s, books were written on them and today, organizations such as the American Society of Training and Development and Training Magazine’s Top 100 use the models as critical pieces to their objectives.

Readers of this article will probably be able to cite Kirkpatrick’s 4 Levels of Learning Evaluation, lets review them briefly just for fun.

Level One: Reaction. This answers the question, ‘Did they like it?’ If done right, it can be a set of performance measures for all the pieces that comprise the satisfaction component of a learning program.

Level Two: Learning. This answers the question, ‘Did they learn?’ Since training is in the business of transferring knowledge and skills to their customers it is only fitting to measure performance in this area. Just because someone had a satisfying experience does not mean they learned new skills or knowledge.

Level Three: Behavior. This answers the question ‘Did they use it?’ A learner could have learned significant new knowledge but not been given the opportunity to apply the training on the job, hence it is wasted training as such this is a key performance metric.

Level Four: Results. This answers the question ‘Did it impact the bottom line?’ Probably the most important measure to a CEO or anyone outside of training. It really looks at the business results changed by training.

The Kirkpatrick model is nice, but without a process to really measure these levels it might not be practical. Enter Dr. Jack Phillips. Phillips is another individual you’ll run across when doing research for your measurement solution. He has had two major contributions to the field of learning measurement that will help you in your learning measurement solution. First, he built a process to measure the Four Levels. Second, he added a fifth level, ROI. Combined, his ROI Process is used around the word as a tool to help learning organizations measure Kirkpatrick’s Four Levels, and Phillips’s fifth. You can see a brief overview of the Phillips’s ROI Process in Exhibit One.

As a way to ensure your measurement process is accepted by your team and by management, leveraging over 80 years of industry accepted models such as Kirkpatrick’s Learning Levels and Phillips ROI Process is a smart move. It is significantly easier to get buy in from stakeholders by suggesting a solution that is industry accepted as opposed to something you drew up on the back of a napkin during your lunch hour.

But, you’re not out of the woods yet, not by a long shot. If you read the books by Drs. Kirkpatrick and Phillips or attend certifications on the ROI Process endorsed by the ASTD, you’ll quickly conclude that it is a lot of work. After all, nothing that is worth doing is easy. The ROI Process has been accepted and works because it is rigorous, conservative and solid. The Learning Levels have been embraced by so many because of their unique balance of performance metrics ranging from satisfaction to linkage to business results. So putting what is right into practice is the next step but if it takes a significant investment of financial, physical and human resources, that is not something you can easily sell to management. Asking for more money to measure training than the cost of training itself would not be prudent. So what can be done?

Enter technology. The new millennium, if nothing else, has taught us how to leverage technology so we can do more with less. The next section will discuss how to use technology as an enabler to practically implementing industry accepted measurement models.

Technology Wrapped Around Methodology:

Any measurement process aims at collecting data, storing it, processing it and reporting it. That is what needs to happen to measure Kirkpatrick’s Four Levels, Phillips’ Fifth Level and to work your way through the Phillips ROI Process.

With this in mind, why not leverage technology to streamline the steps so you don’t have to see significant outlays of resources for your measurement solution? Let’s start with data collection. There are a host of inexpensive web-based data collection tools. These tools are easy to use and allow you to collect data from learners, instructors, managers, and any stakeholder at any point in time. Leveraging the Internet to collect data saves costs of paper processing. It is worth investigating and leveraging where practical and feasible. Even if paper is necessary, scanning technologies can easily scan quantitative and qualitative data into databases for centralized data storage and processing.

When it comes to data storage, I don’t mean that metal file cabinet in the basement closet of your office building that so many end of class evaluations go to as their final resting home. Simple tools like spreadsheet applications can be inexpensive and powerful technologies where data collected online can be dumped for efficient data processing. More sophisticated solutions include using local relational databases such as Access that is more powerful than Excel or using enterprise relational databases such as SQL or Oracle that can span the organization easier. Data storage in technology tools is central, secure, and makes processing easier than that basement file cabinet.

Data processing is probably the most time intensive piece in all of measurement. The key here is to have flexible tools that allow for the building of queries that can then be automated or standardized. OLAP tools such as Cognos or Microsoft Analysis Services are very powerful for querying large amounts of data. However, a strong word of caution. Do not throw technology into the hands of the functional users untrained on learning measurement. Allowing a line of businessperson to write their own OLAP query that compares learning data to business data can be counter productive if not dangerous if they do not know what they are doing.

A have to pause in our story to provide an example here. I read a recent article of a sales department reporting an ROI on training at over 10,000%! At first glance that is phenomenal. A deeper discussion revealed that there was no attempt to isolate a revenue increase to training, it just so happened that revenues increased at the same time sales training occurred. The data showed this correlation and hence the training department attributed the sales rise to the training. What about the economy, the competition, technology, people, process and a host of other factors that could have accounted for the sales increase? This was a classic case of uneducated users being handed a data processing tool to write their own queries. Not something I would suggest doing.

Back to the article. If you have the right people writing appropriate queries that a novice user can then run reports against, you can leverage OLAP tools appropriately. In doing so, you can help a diverse set of stakeholders by creating queries that result in reports targeted at the right level of person for the analysis.

This leads to reporting. In most organizations there is a need for 4 primary types of reports. Your measurement solution needs to be able to provide reports for each. These reports include tactical, aggregate, executive, and value analysis.

Let’s start with tactical reports. These are for the staff personnel in the training department that need to manage the learning investments on a day-to-day basis. Examples of tactical reports include class evaluation summaries, respondent qualitative verbatim, and the actual evaluations themselves. These are used to spot problems before they can get any bigger.

Next are the aggregate reports. Middle managers in a learning group use these reports. They aggregate tactical data for monitoring and quality control. For example the person responsible for managing the sales curricula needs to quickly view all the course titles and determine which are most effective. The person managing the eLearning content needs to easily see how effective it is compared to traditional instructor-led training. These managers need to see tactical data rolled up and they need to be able to filter it down too. Reports for these folks should do this.

Then executives in the learning group need reports. They need to see comparisons internally and externally. For example how did employees in each line of business that fund the training see the training impact their jobs? How did the key indicators of training compare to external benchmarks? And what exceptions occur on a day-to-day basis? Executives need summary, exception-based data that is comparable internally and externally.

Finally there is value analysis. This is packaging it all together. It is your balanced scorecard that you can present to management when the budget is up for renewal or you need to fund a new program. It’s the measures you have an appointment with CEO to review on a quarterly basis cut by eLearning vs. ILT so the CEO can see how that eLearning investment is doing. Value analysis sums it all up. It presents all the measures in an easy-to-generate and easy-to-interpret scorecard. Value analysis allows a chief learning officer to sit down with a C level person or a line of business manager to cover how training impacted the job, how human capital was improved through training, and yes, the financial return on training in hard and soft dollars. Metrics such as benefit to cost ratio, payback period and ROI percentage are in the value analysis.

Examples of some of the reports mentioned above can be found in Exhibit 2 and 3. Exhibit 2 shows how actual data can be compared to performance goals and external benchmarks for each level of learning for continuous monitoring of the Learning Levels. Exhibit 3 shows the balanced scorecard that can be presented to management to showcase value in learning investments.

A Solution to Work 100% of the Time:

We’ve talked about how to take solid models by Kirkpatrick and Phillips and wrap technology around them for your measurement solution. But, you need to ensure that your solution has flexibility. Most of the time you can get by with reasonable indicators to manage your business. These don’t have too be in depth and dead on accurate but provide enough confidence to allow people to make intelligent decisions. Designing the right data collection instruments targeted at the right stakeholders at the right time is important to building a model that is easy to use but provides reasonable outputs. Recognize that you might want to go deeper when the program is strategic or cost a lot. Thus your solution should allow for more in depth measurement to occur.

Because of this need for flexibility, we have devised a three dimensional model that can help provide measurement solutions that work with 100% coverage. These models are the learner-based, manager-based, and analyst based solutions. You get good breadth in measuring all 5 Levels of learning measurement in all of these models. But, the depth increases as you move from learner-based to base to analyst-based. The cost and complexity of the measurement solution increase too, so be careful of that.

Learner Based:

A measurement model that captures data from training participants at two distinct points during the learning process. The first point is directly after the learning intervention (Post Event) where the main measurement focus is on Kirkpatrick’s Level I – and Level 2 to gauge satisfaction and learning effectiveness. Because there is a high response rate to these data instruments it is also critical to capture indicators for advanced levels of learning such as Level 3 – Job Impact, Level 4- Business Results and Level 5 ROI. These indicators are in effect forecasting or predicting the future impact the training will have on the participant and the organization.

A second data collection point is a follow up survey conducted a period of time after the participant has been back on the job. This survey is meant to true up the forecast and predictive indicators of Levels 3, 4 and 5 by gathering more realistic estimates now that the participant is back on the job.

The approach is low cost if one leverages standard data collection instruments across their training and utilizes technology and automation to capture, process and report the collected data. Thus it can be used for all of your training, each time a participant takes a class to yield continuous measurements.

Manager-Based:

This method has the same data collection points as the learner-based solution but adds a manager-based dimension. The manager of the participant attending training is another important data point. They can be sent an evaluation instrument timed when the participant receives a follow-up. The manager survey focuses on Levels 3, 4 and 5 of the Kirkpatrick and Phillips models therefore getting estimates surrounding job impact, business results and ROI from the manager’s perspective. The manager survey also asks ‘support’ type questions to understand the on-the-job environment where the participant applied the training.

Due to the increased effort it takes to conduct and analyze manager surveys the cost and time to measure at this level is higher than the Learner-Based approach. But, with automation and technology to facilitate the dissemination, collection, processing, and reporting of the data, the cost and time can be minimal. The result is that it could be used on a continuous basis for every training event a participant attends. More realistically, it will be used on a periodic basis for more strategic programs where manager data is more relevant.

Analyst-Based:

This approach uses significantly more comprehensive post event, follow up and manager surveys it also uses other analytical tactics that go beyond surveying. For example to analytically measure Level 2 – learning effectiveness a detailed test is designed and administered to participants. Due to the time commitment of conducting a significantly detailed data collection and analytical exercise the Analyst-Based approach is only used for about 5% of all training programs in the organization. Typically these programs are the more strategic or visible and have the budget to afford a more costly and time-consuming measurement exercise.

See Exhibit Four for an visual of the learner, manager and analyst based methodology.

Concluding Thoughts:

Activity measures are necessary to manage a training business and will tell you how much you trained. But, the workplace has changed and the demand for more comprehensive measurement tools heightens the need for better use of performance metrics that tell you how well you trained.

Through appropriate methodology that is made scaleable, practical, and replicable leveraging technology, organizations can provide performance metrics in a cost effective manner. Organizations should always keep in mind that you do not have to have the most perfectly accurate quantitative metrics in order for them to be worthwhile. In fact, a recent study published in Harvard Business Review (May 2003) found that senior managers make decisions on other instinctive factors not the highly accurate and highly costly data they are provided from highly paid number crunchers. Use reasonable data based on indicators that are meant to predict and estimate your key performance metrics. This will save you significant cost, significant time, and accomplish the objective of providing performance measures to management so they have the right metrics for their information decision-making.

Continue Reading

Republished from CLO Magazine, October 2003

By Jeffrey Berk

A top question often asked of organizations creating and revising processes for learning measurement is ‘What are the best practices?’ This question is asked more and more because training departments are being asked to prove their value more and more. In tough economic times all aspects of the organization are being asked to justify their existence, not just training. However, the more prepared training organizations can be to respond to such inquiries the better likelihood they have of maintaining a value added role within the company. Continued

Continue Reading

Republished from T+D Magazine September, 2003 (online issue)

By Jeffrey Berk

If you’ve been following the learning industry recently you’ve probably heard the term ‘learning analytics’ thrown around. Learning Analytics is the new buzzword for the process by which learning professionals analyze critical indicators within their business to not only continuously improve but to demonstrate value to stakeholders and make better decisions to optimize learning investments. It is not just about cool technology. Learning professionals should be taking a very close look under the hood of the ‘analytics’ tools to ensure they are gathering the right data and leveraging the right queries that senior managers need to understand the value of learning investments.

The term analytics is part of a technology term commonly used as the engine for the complex and heavy number crunching that takes place in powerful measurement solutions. That technology term is referred to as OLAP – online analytical processing. OLAP tools are rapidly becoming the tool of choice to aggregate and parse through thousands of data points to serve up relevant metrics surrounding the learning organization.

However, it is extremely important to point out that OLAP tools, although powerful, are only as mighty as the queries they execute and the data within the queries. The most challenging elements of learning analytics has never been and will never be the technology, but instead are the processes that work behind the scenes to feed the technology.

First, the source data is absolutely vital. Feeding powerful OLAP tools the right data is extremely important. Activity based data such as number of employees trained, number of classes run, attendee to enrollment ratios etc. are nice forms of data. And, when combined with financial information fed from ERP systems, such as organization revenue or a training budget, they can make for interesting measures.

However, a senior manager who funds organizational learning may not find activity metrics all that meaningful. An activity metric such as number of employees trained per sales dollar doesn’t tell the manager how a multi-million dollar training investment impacted the job performance of the employees trained. Thus, you need to power your OLAP engines to collect performance-based data. This is data on learner reaction to training, quantitative evidence of knowledge/skill transfer, key ratios around impact to the job as observed by managers or experts or evidenced through participants when they are back on the job, linkage of training to key business results, and finally the financial return isolated to training relative to the cost of deploying the training. These performance metrics are significantly more powerful to proving the value of learning versus activity metrics. They are also harder to collect and many learning organizations don’t collect them because of the perceived complexity of doing so.

Unfortunately, the day is gone when the learning organization can shrug off performance-based metrics. Sluggish economies are prompting management to scrutinize training budgets. Performance metrics are quickly becoming a learning organizations best (and sometimes) only friend when they want to be at the table with management helping improve the business as opposed to on the table susceptible to cost cutting.

Building the right processes, using best practice measurement methodologies, and leveraging appropriate technologies can help the learning organization significantly streamline the cost and time to collect, store, process, and report performance-based data gathered from multiple stakeholders and systems as opposed to just activity data that is easy to pull from an ERP or LMS.

Once you have the best performance-based data to run through OLAP tools you then need to design the most optimal and meaningful queries so it can be leveraged properly. Here again, OLAP tools are wonderful enablers but without the knowledge of measurement and learning the query you create can have relatively little value. Solid queries based on tried and true learning measurement models such as Dr. Donald Kirkpatrick’s Learning Levels model that has been around for nearly a half a century and Dr. Jack Phillips ROI Process that is widely used around the world and endorsed by the American Society for Training and Development, are vital connecting points for OLAP queries. Programming a balanced scorecard of key performance indicators tied back to performance-based data linked to industry acknowledged measurement models are the queries you need to run your learning organization. Simply providing an OLAP tool and providing your stakeholders the flexibility to write their own queries can be dangerous and counter-productive.

Learning analytics is a powerful term and not one that should be taken lightly. The learning industry is on the verge of revolutionary change through the use of analytics. In other business processes such as the finance function, supply chain management, and customer-relationship management, analytics are common tools used day-to-day for information decision-making leveraging a balanced scorecard of metrics specific to those functions. As the analytics train pulls into the training departments, training practitioners must ensure that the data and queries powering the analytics are the most appropriate for making decisions to manage learning investments. A careful check under the hood of a learning analytics tool to ensure the right data and queries are being used is a best practice and obligation for any organization seeking technology to measure their learning investments.

Continue Reading