Posts Tagged ‘Readiness Assessments’
Republished from CLO Magazine, September 2007
By Jeffrey Berk
This article is about understanding what you don’t know. When you endeavor to try to know what you don’t know, we call that a readiness assessment. We’ll spend most of the article discussing readiness assessments from a learning and development perspective as it relates to measurement, evaluation and analytics. However we devote some space near the end of the article to discuss the larger talent management and human capital elements of analytics.
What is a Readiness Assessment?
A readiness assessment is a planning tool to help identify gaps in existing strategies and processes. A readiness assessment accomplishes three goals:
1. Guidance for what you don’t know about learning analytics
2. Aid to building a learning measurement strategy
3. Documents gaps in the current ‘As is’ state versus the future ‘Should be’ state
The readiness assessment is administered using a focus group or interview or accomplished self-sufficiently via a survey. Interviews are optimal because the interviewer can use the one-on-one interaction to drill down into specific components and seek out details and examples. The goal of the interview is to thoroughly understand the ‘As is’ state.
There are four main components to a learning measurement readiness assessment. The following will be discussed in the proceeding paragraphs:
1. Stakeholders
2. Strategy
3. Process
4. Technology
Stakeholders
Stakeholders represent the consumers of learning metrics. These may include the following:
* Instructors
* Instructional designers
* Organizational development managers
* L&D Vice President/ Chief Learning Officer
* Line of Business manager
* Executive management
Understanding stakeholder needs is a way to identify the types of metrics that these consumers are interested. Some key questions to ask the L&D organization with respect to this audience include the following:
1. Do you know who the key stakeholders for learning metrics are?
2. What specific metrics are these stakeholders looking for?
3. Are there specific reports available to meet their needs?
4. How satisfied are these stakeholders with the timeliness and usefulness of learning metrics today?
Further, the assessment should tackle the metrics balance as different stakeholders have diverse needs. The assessment should look at the following four broad areas to assess stakeholder needs:
1. Operational data: Is there a need for activity data (ex. Number of students trained)
2. Performance data: Is there a need to show how well you train as opposed to how much you train? (ex. Satisfaction scores, or linkage to business results)
3. Financial data: Is there a need to show fiscal stewardship? (ex. L&D budget to actual or L&D budget as a percent of payroll)
4. Cultural data: Is there a need to cultivate an environment that supports learning and finds it strategic? (Ex. How many external awards or internal recognition points has L&D received?
Finally, the assessment should be probing to ensure the metrics the stakeholders desire will be presented in a way that leads to timely and useful data for information decision-making. This is known as actionable data. There are four ways to achieve actionable data that should be investigated during the assessment:
1. Benchmarks: Do stakeholders value internal and external benchmarks to use as points of reference in interpreting their own data?
2. Goals: Can L&D establish challenging yet attainable goals to show actual performance against stated goals?
3. Trends: Can L&D measure consistently and repeatedly to derive trends so stakeholders know if the metrics are moving in the right direction?
4. Color coding: Do existing stakeholder metrics present information in a manner that is easy to read with a red/yellow/green analysis to quickly pinpoint opportunities?
Stakeholder analysis for learning metrics is important because they are the users of the information. Success is more certain if aligned with stakeholder desired outcomes.
Stakeholders
Strategy is developing the approach to learning metrics. This is reviewed in a learning measurement readiness assessment to understand the methodology and functional design of the learning analytics process.
Some key questions to ask during this phase of the assessment include the following:
1. Do you have a strategy? Is it practical to implement?
2. Does the strategy focus on impact and ROI?
3. Does the strategy consider benchmarking?
4. Does the strategy consider standards and consistencies?
5. Does the strategy provide meaningful information in a timely manner?
It is challenging to build an analytics policy or process without an overlying strategy. Hence the importance of this section. If key decisions on what to measure and how to measure and when to measure are not answered, the following risks will be more prevalent:
* Learning will be measured inconsistently
* Learning will not be comparable across the organization
* Duplication of effort is likely to occur
* Additional cost is likely to be incurred
* Timeliness of providing metrics will likely be prolonged
Due to these risks the organization must have business rules or standards for learning measurement strategy. Some core rules might include the following, and should be included in a measurement strategy:
1. The key performance indicators for the L&D organization
2. The measurement protocols or standards (especially for evaluation)
3. The linkage to business strategy
4. The linkage to credible measurement methodology
5. A process to execute the strategy
Process
Process represents development of the inputs, activities, and outputs of learning measurement. This is reviewed in a readiness assessment to understand the physical and financial resources involved in producing the metrics. The assessment should ensure the process is practical and repeatable given existing financial, physical and human resources. The assessment should also ensure the process, as stated, is functioning as designed.
Some key questions to ask during this stage of the assessment include the following:
1. Are key performance indicators monitored regularly?
2. Do you evaluate 100% of learning events?
3. Is there a formal budget for learning measurement?
4. Are there dedicated resources for learning measurement?
5. What percent of time is administrative vs. value added?
The key areas of concern, those to drill deeper with the interviewee, would be the following core elements of measurement:
1. Data collection
a. Look at the key performance indicators and ensure a balance exists (operational, financial, cultural, performance)
b. Study the evaluation instruments and ensure they consistently collect the right data
c. Understand the volume of learning to ensure collection is scalable
d. Review technology tools used in the collection process for efficiency
2. Data storage
a. Review the database(s) and look for a central database or a way to consolidate data into a data warehouse
b. Understand the security surrounding the database(s)
c. Determine how easy it is to extract data out of the database(s)
3. Data processing
a. Determine if there are self-sufficient query capabilities
b. Review how data is aggregated and filtered
4. Data reporting
a. Review the standard reports given to stakeholders
b. Study reporting functionality and features
i. Benchmarking
ii. Trend lines
iii. Statistical sophistication
iv. Drill down capabilities
v. Executive summary capability
The key is to recognize that the four activities above are administrative in nature, and therefore non value-added. The resources allocated here should be minimized by having solid business rules, standards, templates, and technology. The goal is to get to reports quickly so that users can use them for analysis and decision-making. Hence we need to study the learning measurement process carefully during a readiness assessment.
Technology
Technology is the last, but not least, component of a learning analytics readiness assessment to consider. Technology is studied to understand where manual tasks can be automated and to help streamline administration.
Some key questions to ask at this stage include the following:
1. What percent of data is gathered online vs. paper?
2. Is there a central database for storage?
3. Is there an OLAP (online analytics processing) database for custom querying?
4. Is it easy to drill down for tactical analysis?
5. Are results available real-time through self-sufficient tools?
There are areas of overlap between technology considerations and process portion of the readiness assessment. This is perfectly acceptable. The process area looks at transactional flow, whereas technology looks at the mechanisms to produce the transactions within the flow and their degree of technology sophistication. For example, use of a basic internet survey tool can make a world of difference in data collection as opposed to manual paper processing. In addition, a SQL database may have optimal central storage features over multiple worksheets in Excel. Further, OLAP tools for querying the database for processing may lead to self -sufficient users versus IT department involvement when a query needs to be done.
In today’s world, technology should be considered throughout the process. Technology is inexpensive and powerful, it can minimize administration, leading to a shift to maximize time spent on analysis, communication and improvement.
Broader Human Capital and Talent Management Implications
As stated, we’ve focused most of this article on the readiness for learning measurement. Yet L&D is increasingly asked to play a broader role in the talent management and human capital management area. How are these areas measured? What makes a complete set of processes for these areas? These are questions many organizations are struggling with right now.
1. Manage Deployment of Personnel
a. Forecast Workforce Requirements
b. Recruit, Select, and Hire
c. Succession Planning
d. International Assignment
e. Mobile Workforce
f. Employee Turnover
2. Manage Competencies and Performance
a. Competency Management
b. Performance Appraisal
3. Develop and Train Employees
a. New Hire / On-boarding
b. Learning and Development
c. Coaching and Mentoring
d. Leadership Development
e. Knowledge Management
4. Motivate and Retain Employees
a. Compensation and Benefits
b. Employee Satisfaction
c. Employee Engagement
d. Work/Life Balance
e. Workforce Diversity
From the above scheme you can identify the all areas of human capital process measurement in a complete manner. From this point you can then use diagnostic tools to assess the readiness of measurement sophistication within each to determine where to focus efforts. Most of these processes need better management, and measurement is a start to getting a handle on the management of them.
Conclusion
A learning measurement readiness assessment, and in broader terms, a human capital or talent management readiness assessment, can help diagnose gaps in existing proceses and pinpoint opportunities for improvement in a prioritized manner.
In order to perform the assessment, it may help to have someone who not only knows the domain (L&D, human capital, talent management) but also is an experienced interviewer or facilitator of focus groups. Finally, from a change management perspective, to avoid political agenda’s and ensure everyone provides objective responses, a third party may be the best option for leading these assessments.
We hope this has provided some background and context for the learning readiness assessment and how it can be a critical tool in your learning measurement endeavors.
