Metrics that Matter® Learning Analytics at a Major Financial Services Company
Company Overview: The organization is in the financial services industry and is one of the top 25 largest banks in the United States. The company employs over 15,000 people and has over 1.5 million customers.
Description of Learning Program: The Learning and Development Team of the financial services organization supports the retail banking and mortgage business units comprised of 8,800 employees at 600 locations. The team offers over 100 training courses
Measurement Approach: The approach of the financial services organization was to proactively show the effectiveness of learning programs. A small-scale pilot was initially done with Metrics that Matter®. Surveys were created that tied learning to business strategy and were modified based on learning delivery and to accommodate learners with limited access to technology.
The approach included the use of Post Event (end of class) evaluations to measure quality and predict effectiveness. Follow-up instruments were deployed to measure impact on the job. Tests were conducted through Metrics that Matter® to validate knowledge transfer and skill competency.
The financial services organization focused on educating the workforce and the users of the analytics technology on why and how this process and technology functions. The process was integrated into the course manager’s area of accountability so it had ownership.
Measurement Obstacles: Many employees work at bank branches without access to email, preventing them from receiving end of class surveys electronically. Paper is often used to collect the data and the financial services organization was able to outsource the processing of the paper to KnowledgeAdvisors to save time and cost. Further once processed, the organization can view reports without having to perform calculations on raw data inputs. Both paper and electronic evaluations are stored and reported in a single database therefore allowing for a centralized reporting access.
Measurement Successes:Several reports are used by the Learning and Development team. Qualitative comments can be queried and reviewed to pinpoint success and improvement opportunities. Quantified calculations of questions and answers that display the variance in the data as well as the mean score are reviewed on a regular basis to monitor course performance.
A testing summary report is reviewed for each class that has a test. The system automatically grades the tests and identifies participant exam scores. This helps ensure knowledge transfer occurred and helps to improve the course overall.
A powerful report is the Report Card. This is setup to be generated and emailed automatically by the system on a monthly basis. The Metrics that Matter® system sends reports to the course manager and their supervisor. It pinpoints each performance attribute of the training (satisfaction, effectiveness, impact, results and ROI) and benchmarks each course against the internal benchmark database maintained in Metrics that Matter®. This allows management to proactively manage the operation.
To manage both in-house and contract trainers a Comparative Performance Report is used to show the quantitative scores of each instructor against each other and a benchmark. It ensures consistent quality.
A summary scorecard that is specifically meant to show value to clients and stakeholders is run when key programs occur. For example a key program was a customer service focused new employee orientation. The report showed quality ratings above 90% and 85% of attendees felt the program had a significant impact on customer satisfaction.