Technological Considerations in Learning Analytics
CLO Magazine, March 2005

By Jeffrey Berk and Scott Magee of KnowledgeAdvisors

Learning Analytics technology is an enabling tool that can significantly assist learning organizations in understanding how to better train and develop employee’s partners and customers. The building blocks of learning analytics technologies include the way analytic data is 1) collected, 2) stored, 3) processed, and 4) reported. Mapping out the points of automation for each of these is helpful when designing a learning analytics technology.

Learning Analytics technology is an enabling tool that can significantly assist learning organizations in understanding how to better train and develop employee’s partners and customers. The building blocks of learning analytics technologies include the way analytic data is 1) collected, 2) stored, 3) processed, and 4) reported. Mapping out the points of automation for each of these is helpful when designing a learning analytics technology.

Surveys are easy to scale. Items on the survey should conform to the metrics of interest—and not just collect data in a vacuum. Instead, the survey should collect what is needed. Another benefit of surveys is that they can be automated, freeing up time for analysis based on the data.

Data Collection

Data collection is how the organization obtains its original data for storage, processing and reporting in the analytics tool. A major technological consideration in data collection is to leverage online and email data collection devices. Yes, response rates may drop but timeliness of processing and reporting will go up and costs will be saved. Eaton Corporation’s Eaton University is a great example of technology for data collection. According to Joyce Gilman, Operations Manager of Eaton Corporation, “All data for the University is collected via email for end of class, follow-up and manager evaluations. This saves us significant time and cost while continuing to provide Eaton University a significant amount of data to analyze for continuous improvement opportunities.”

But what if your organization cannot collect the data electronically? Either the culture or the physical infrastructure makes this an impossibility. Scanning technologies can be leveraged to import raw data into analytics technologies. “Although we collect our learning evaluations in a paper-based manner, we have systematized the processing of the data via scanning technologies. The raw data is batch-imported into an analytics tool that processes and reports the data. As a result, our learning organization focuses on the results and quality of the training, not compiling numbers,” commented Robin Killeen, Project Manager for Discover Card.

When collecting data it is important to be mindful of data security as well as data privacy and protection. Physical security controls should be in place to prevent unauthorized access. Network security is equally important, ensuring that firewalls are in place to prevent unauthorized access. Data privacy and protection means that respondents have the option to submit data anonymously and they have knowledge of how the data will be used. Conforming to rules such as the Department of Commerce Safe Harbor legislation is a way to provide comfort to users that the data collection site has been certified by a regulatory body as safe in the protection of data privacy.

If the source of collected data is in question it is not worth the pain of readying it for analysis or in corrupting an existing database with bad data. Collecting data used in analysis that has credibility and integrity is important. A best practice company is Hibernia Bank. According to Robert Baer, Design Team Leader, Hibernia’s learning organization knows exactly the format and source of real business results produced by various systems and technologies. “We use this data to accurately and credibly link the results to training programs. This makes the analysis well received by our senior management.”

A final technological consideration is the configurations and settings. Establish minimum configurations for browser and operating systems that are commonly used within your organization and business in general. For example only supporting Windows XP when the majority of your workforce is on Windows 2000 is not a flexible minimum supported option to rollout to your workforce.

Data Storage

Data storage describes the environment the organization uses to maintain its learning analytics data. The data structure and use becomes a significant technological consideration here. Really knowing how the data will be stored and what its use will be is critical. GTECH, a leading provider of gaming and technology solutions worldwide, has a detailed outline of how its training courses fit into hierarchical curricula and programs. Margaret Lamb, a GTECH Instructional Designer, said, “Knowing that our analytics technologies can support how we organize our curricula in its data structure is very important. This means that we can measure segments of our course offerings, as well as the total program. This strategy really helps us understand and manage our learning programs.”

Centrally storing all of your collected data in a single data warehouse is critical to the processing and reporting elements of analytics. Large IT hardware and software vendors often train high volumes of end customers through third-party partners around the world. Many leverage a third-party learning analytics tool that acts as a clearinghouse to gather the data from multiple decentralized locations and then centrally stores it in a single database. The central storage of the data then allows these vendors to access all of this data easily and in a timely manner to better manage quality, performance and value aspects of their training.

Analytics tools rely on large volumes of data to allow users to slice and dice in multiple ways. Each data point must be stored in its raw format in the database. This means that basic tools like a spreadsheet application may have size limitations on data storage. It makes sense to therefore store the data in more powerful tools such as OLAP systems that are great are powering through large sets of data. New Horizons Computer Training Centers is a great example. This professional learning organization has collected over 1.5 million evaluations in the past few years. When responding to a recent RFP, they were able to quickly query the OLAP database and find out that the prospective customer had received training from New Horizons over 100 times and in 30 locations around the world. This made for a more impactful response to the RFP and the data was gathered in minutes.

Retention of data coincides with storing it. Over time, large amounts of training data can consume even the most powerful databases and slow them down. Establish a data retention policy early in the analytics process and communicate it to users of the analytics technology.

Finally, technology comes with the risk that data can be lost or corrupted. In order to mitigate this risk, establishing backup procedures is one technology consideration not to be overlooked. An hourly or more frequent transactional backup of data is important. Nightly full backups are recommended. Ensuring that the backups cannot be overwritten is a further precaution. Finally, taking the backup files to offsite locations is a physical security procedure.

Data Processing

Data processing describes the actions taken on the stored data to transform them into business intelligence ready for analysis. A primary element of processing is to address data formatting issues. Understand the native format and origin of data before including it in an analytics engine. This will help in determining the right option for exporting and manipulating the data. For example, data in a Relational Database Management System (RDBMS) is easy to de-normalize and analyze in other RDBMS’. But data being imported for processing from a static Excel spreadsheet may be more challenging to upload and analyze in an RDBMS. The car maker Audi worked closely with technologists to appropriately format data derived from these systems to provide the analytics technology a common file feed. According to Annette Eagle-Dull of the formerly of the Audi Academy, “This file feed is now being customized for import into an analytics tool. Once done Audi will be able to analyze their data in a single platform with multiple filter options and make the analytics tool accessible to their instructors and designers for quality analysis.”

OLAP capabilities were discussed a little bit in the data storage considerations but are equally important to discuss in processing considerations. Using sophisticated Online Analytical Processing (OLAP) tools to process data increases the power of analysis. There are a myriad of OLAP tools on the market that range in price and functionality. The key element here is to build the right structures within the OLAP tool to allow a functional user to easily manipulate the data self-sufficiently. This often requires front-end interfaces to be built on top of these tools eliminating the need for IT departments to use tools like this to write reports each time a user needs them. Nonetheless the power of OLAP tools is important in a user ability to ‘slice and dice’ the data by key learning and organizational elements. Examples of filters in the learning space include:

  • Instructor
  • Course
  • Class
  • Curricula
  • Program
  • Location
  • Learning Delivery
  • Start Date/End Date of Training
  • Type of Evaluation
  • Question/Question Category
  • Business Unit
  • Customer/Client

A leading professional services firm’s training operation views its learning metric data through an internet-based OLAP tool that has the capability to compare performance across multiple attributes of its learning operation and its business operation. According to a director the firm’s global learning, queries can be run by filtering on course, instructor, and delivery components of training but also by line of service, sub group and industry type which help the learning organization understand the impact training has not only on training attributes but also on key components of the business.

Netxel Communications, Inc. leverages the technological consideration of processing consistency in its analytics tools. Per Danny Brown, Manager Evaluation and Metrics, “All learning metric data is processed from a single point of origin. This is made possible by the analytics technology’s preconfigured queries that reach back to the stored raw data to perform queries not calculating from previously calculated data which can skew results.” Brown articulates the importance of relying on analytics tools that conform to processing consistencies. “You need to be comfortable that the technology will always run the same query in the same way no matter what the data set or time frame is going to be,” said Brown. Ensuring the queries are preconfigured across the technical architecture of the analytics tool to be consistent is important to system integrity.

Technologists will need to understand the syntax languages of the various analytics tools that are being utilized to store and process the data (Access, SQL, Excel, SPSS, SAS, etc). Resources will be needed that are knowledgeable regarding ETL data (Extract, Transform and Load). ETL is a data integration function that involves extracting data from outside sources, transforming it to fit business needs, and ultimately loading it into a data warehouse. This is where data is transformed from one native language to another so that it can be processed and analyzed in an analytics tool. If technologists are not well versed on ETL and syntax there is a higher risk of misinterpretation and use of data types in the analytics tools.

There are various feeder systems to a learning analytics tool. HRIS and LMS systems are the two primary feeder systems that supply data to a learning analytics tool. Leveraging a standardized XML schema can allow the extraction of data in one system and import/storage of that data into another in an automated and friendly manner. For example, Caterpillar Inc.’s learning organization used a standard schema to export learner demographics and class attributes housed in the LMS to their analytics technology tool. According to Veronica Schmeilski, the Caterpillar project manager on the integration, “The IT organization was provided a standard XML schema by the analytics vendor to extract details from the LMS (registration, completion, and demographic data). Nightly this data is sent via FTP to the analytics vendor where it is automatically picked up and imported into the analytics tool that then initiates evaluation collection, storage, processing and reporting protocols.” Overtime the schema changes have occurred as the business and learning evolves to make it further customized to Caterpillar’s needs.

Data Reporting

Data reporting describes the formal presentation of the results of a processed request. A major technological consideration is ease of use. This may seem functional rather than technical. But, ensuring the functional users can see results in a manner that is easy to interpret is important. Using tables and charts with appropriate language surrounding the results mitigates risk of misuse. Investing in charting software or other packages that can appropriately display results is important. Defense Acquisition University (DAU) uses easy to read charts (red/yellow/green – stoplight charts) that any report user can quickly understand to identify strengths and areas of opportunity. Mark Whiteside, DAU Director of Performance and Resource Management ensures that easy-to-interpret reports can be run across multiple areas of DAU training such as by instructor, location, course, and delivery. Per Whiteside, “We want to maintain our position as a world-class learning organization. Ensuring we have data that can be quickly interpreted on the key elements of our quality process is critical. The format of our stoplight-type reports makes them easy to evaluate performance in an efficient and effective manner.”

Quality in the learning analytics technology ensures credibility in the technology. Changes when needed to the reporting environment should first be done in a safe test site to ensure the change is appropriate then migrated to the production site, the official reporting site. This is done to ensure functionality and security is not adversely affected. Also regression test all new code releases prior to implementation into the reporting environment. This means test prior functionality even if not presumed to be affected by new functionality. This minimizes negative impacts new functionality has on existing functionality.

Aligned with the quality process is communication of maintenance windows and escalation paths for customer support. If the reporting application needs planned maintenance communicate the scheduled downtime to users of the application in advance and make the change takes place during the least obtrusive time. Support the reporting application with technical help. It is advisable to establish appropriate escalation paths that define the levels of support. For example, Tier 1 support could be basic technical glitches that are not impacting all users and to which there are workarounds. A more significant escalation could be an application outage affecting all users. The owners of the learning analytics tool should establish a supporting process so users of the learning analytics technology have a means to report and receive feedback and have the appropriate support points of reference depending on the technical problem.

Concluding Remarks:

Learning analytics has emerged as a powerful business decision-making tool for learning operations managers. If built with the correct technological considerations these tools can offer timely and accurate business intelligence to help provide answers in terms of learning activity and performance.

Grant Thornton’s corporate university understands the power of metrics and the technology considerations to make it possible. Bob Dean, Chief Learning Officer for GTU (Grant Thornton University) established the cornerstones of linking training to business results and showing value as part of his learning strategy. Dean, a visionary in this field, leverages virtually all of the considerations discussed above. “At GTU we want to focus on proving the value not processing the data. So we collect all of our metrics online. We have integrated our analytics system with our learning management system so we centrally store our analytics data and let the analytics tool process it. We then leverage the data when making courseware decisions and in showcasing the power of training to our management.”

The key with any analytics technology is ensure it can automate as much of the data collection, storage, processing and reporting as possible. Doing the automation in a way that makes learning analytics practical, reasonable, cost effective, and repeatable is the benchmark for successful implementations of this technology.

Jeffrey A. Berk

Vice President, Products and Strategy, KnowledgeAdvisors, a learning analytics technology company

Jeff led the Benchmarking Group at Andersen prior to joining KnowledgeAdvisors and he brings a deep level of expertise in measurement. He is responsible for designing and implementing the suite of products and services for the company