Case Study 1: Real Estate Company
When the Microsoft Office suite became the standard across a real estate company, there was a need to provide training to all employees on the most frequently used Microsoft applications—Outlook, Excel and Word. The company briefly considered outsourcing the training to an off-site training provider, but it was concerned that the costs for travel and time away from the office would require a much larger budget than an e-learning solution.
In the end, the company chose an e-learning approach and then proceeded to find the right vendor to meet its needs. Implementation of the solution incurred both technical labor and non-technical labor costs. The technical side of implementation required only a few hours of investment. The company invested significantly in non-technical labor costs for a variety of activities, however, including overall program development and program marketing within the organization. These non-technical costs represented approximately 67 percent of the total first-year budget. But once the program was launched, ongoing costs were minimal, with ongoing program coordination requiring the work of only one person for an hour or two a week.
Within the first eight months of the program, more than 150 courses were completed in full, and more than 500 courses were completed at least in part. In general, the employees were excited to have an opportunity to receive training in the software that was a critical part of their workday. The company believes that this increase in employee satisfaction had a positive impact on employee retention.
The return on investment was calculated by making conservative assumptions regarding the benefits received. Although the company realized several benefits from training, only two were selected for analysis—cost savings and employee retention. These benefits were chosen because they represented the greatest near-term benefit for the company. The benefit values were also weighted (reduced) to produce a conservative quantification of their impact. Note that the time-to-payback (five to six months) is the result of estimating “straight-line” costs and “straight-line” benefits. That is, it is assumed that the costs and the benefits are realized at the same rate each month. In reality, this is not likely to be the case. Thus, the time-to-payback is an approximation.
Table 1: Cost and Benefits: Real Estate Company
ROI Analysis
One of the things that stands out in this case is the decision to limit the benefits measured to two items—cost savings and employee retention. The training group did not try to measure too many variables.
The cost of employee turnover represents a tremendous drag on any organization and was likely already being measured. Measuring the before-and-after difference was a practical and low-cost way of calculating ROI. And the positive impact on the organization—as depicted in Table 1—must have drawn the favorable attention of management.
The cost savings of e-learning was an interesting choice for inclusion in an ROI study. The measurement may have benefited the training group as much as it did senior management because it serves as a benchmark for future investments.
The training group’s grasp of the expenses involved was compelling and insightful. Identifying the expense of marketing the program internally was appropriate and showed the group understood its cost structure.
The format for the ROI presentation selected for the research project—the simple graph, statement of ROI and additional benefits—is quickly accessible. It is the type of presentation that, I believe, would be well received by a chief executive officer and a chief financial officer. The inclusion of qualitative benefits is impressive, but not overdone. On the whole, the case is credible and convincing.
It also gives me the confidence that I could do something similar and that the intangibles of measuring the impact of training are not quite as elusive as I might have originally thought.
Case Study 2: Energy Company
Business units at the energy company needed IT technical training for their employees to support new projects and products. Often there was little advance planning, and the training had to be delivered quickly. At the corporate level, employees needed ongoing access to a broad array of technical training subjects.
The company chose an e-learning delivery approach. The company did not want employees to wait for classes or be obliged to sit through entire courses. Also, the company wanted to give employees the flexibility to take as many courses as they needed. Nearly all of the training was delivered via asynchronous, Web-based, self-directed learning. Some employees, however, have begun to discuss what they learn in virtual classrooms. There are only four significant cost areas for the solution: product licensing fees, product implementation, salaries for three in-house part-time program coordinators and administration of course evaluations.
The last 12 months yielded a running average of approximately 3,000 completed courses and 7,000 partial completions. The company believes that the opportunity for employees to complete only a portion of the course was critical for improving employee productivity, which increased because employees spent less time searching for answers to technical questions and had access to training during off-hours. Because employees had answers at their fingertips, they relied less on the help desk and more on the courseware. The result was a reduction in help-desk calls and associated costs.
The return on investment was calculated by making conservative assumptions regarding the benefits received. Although the company realized at least six benefits from training, Table 2 shows that only two were selected for analysis—an increase in employee productivity and a reduction of help-desk call volume. The time-to-payback is an approximation.
Table 2: Cost & Benefits: Energy Company
ROI Analysis
The reservation I have about this case is that I’m not completely clear on how the training group measured employee productivity. Clues are given in the benefits box about greater employee speed to competency, reduced re-work, higher quality of service and less downtime. Are we to assume these factors are the basis of the productivity improvements? With the exception of employee speed to competency, these items are clearly quantitative, and measurements likely already exist. Speed to competency might be easily quantifiable for this company; we just don’t know enough of the context.
I’d be more comfortable with this presentation of ROI if the training group had been clearer on what went into the employee productivity measurement, which is the crux of their case. They could have easily broken productivity into components and raised the credibility of the report, at least from our outside perspective. As this study is presented, it almost looks too good to be true, in my opinion, and has a lower level of credibility than the first case.
Case Study 3: Defense & Aerospace Company
The third case study subject is a telecommunications consulting unit within a Fortune 100 aerospace and defense (A&D) company. The consulting unit builds highly complex information technology systems for its clients and trains clients to use those systems. Although the A&D company provided high-touch technical instruction to its clients, it did not train its consultants—the 70 individuals who were charged with building those systems. The company recognized that to better serve its clients and to stay competitive, it needed to keep its consultants on the cutting-edge of technology.
The A&D company set out to build a robust training program for its consultants. It had to be very selective when choosing course content, as the subject matter is highly technical and changes rapidly. In the end, the A&D company selected best-of-breed content from multiple vendors. This approach was more costly—more than $100,000 per year—but the investment was justified by the value of the content. As the delivery method for the training, the A&D company chose a blended approach—primarily instructor-led classes supported by self-study Web-based content, CD-ROMs and workbooks. The company determined that live, face-to-face instruction would help students gain a deeper understanding of the content. The supporting materials would reinforce what was learned in the classroom. In the end, instructors delivered roughly 60 percent of the content. Students received an additional 30 percent of the content through supplemental course materials. The remaining 10 percent was devoted to individual coaching, which underpinned the entire program. This formal coaching and mentoring process ensured that real performance improvement took place and that the training had a clear impact on the consultant’s professional development. The field service organization reported receiving several distinct benefits from the training, including higher technician motivation and competency and greater corporate productivity.
The return on investment was calculated by making conservative assumptions regarding the benefits received. Although the A&D company realized seven benefits from training, only three were selected for analysis—an increase in employee productivity, product and service quality and customer satisfaction. The time-to-payback shown in Table 3 is an approximation.
Table 3: Cost & Benefits: Defense & Aerospace Company
ROI Analysis
Two things impress me about this case history. The first is the high cost of training per student. Senior management must have had supreme confidence in its training organization and believed deeply in the potential value of the program to fund it. The second is that this company had the data to put a dollar value on customer satisfaction, typically one of those intangibles.
In a slow-growth economy, high customer satisfaction leading to customer loyalty is strategic. As you prepare your own ROI statements, talk with your peers in the sales, marketing and finance organization to determine whether they have been able to pin down a dollar-value correlation with customer satisfaction. If they have, this may be an extremely important ROI factor to measure and to compare before and after training.
A theme running through all these cases is that data already exist in different departments and that critical components of business success like customer satisfaction are being measured. For your own ROI reports, look for these. Existing data that is already being collected may give you an effective way of measuring a before-and-after difference. Knowing what data is available also helps to focus your planning for an ROI report.
Getting Started: Demonstrating Positive ROI From Training
To be fair, most chief executive and chief financial officers simply want assurance that the investment in training has a positive impact on the organization. They are not asking for astronomical return on investment. They are not asking for a 100-page research report. They simply want to know that the training organization has control over its costs and an eye to the bottom line. They will likely be thrilled to see a start made on quantifying ROI. How can you get started or, if started, refine your thinking? The CompTIA/Hurwitz report suggests:
- Start with qualitative data. Most people think of ROI as a number, but in reality, a return can be qualitative, too. For example, level-three evaluations, in which managers report students’ change in performance, can produce highly useful insights for the training manager. This qualitative analysis can serve as the foundation for building a robust quantitative measurement system.
- Conduct ROI case studies. Randomly select a few students or small teams to study. Assess changes in key performance metrics. Collect data by interviewing both the students and their managers and by reviewing data from management systems (e.g., call center data). With this approach, it is possible to project the results (with varying degrees of accuracy) to the larger population of students.
- Start by targeting 100 percent payback. Does your program at least pay for itself? To find out, determine the minimum performance improvement that will yield 0 percent ROI. Sure, achieving payback might not knock the socks off your CFO, but at least it shows that your training program justifies itself. You can use this ROI model to set minimum goals for your training program.
A suggestion not made in the report, but one that would be helpful, is to make course evaluations more business-oriented. Course evaluations have typically been paper-based, asking students to write down their feelings about the training experience. Course evaluations can be transformed by focusing on objective measurements of what is happening in key performance areas at the end of the class, as well as months after the class. Data collection can be facilitated by Internet-based systems, and there are companies that offer these kinds of services.
Although the report mentions certification, it does not explain this measurement in detail. Certification via an examination following a training program is a benchmark of how well the student mastered the body of knowledge. Certified individuals speak the same language and share the same concepts. This helps to enhance productivity. They have absorbed the standards you are promoting through training. Those who do not earn a certificate give the training organization the opportunity to take a more tailored approach. You have effectively identified those individuals who may learn differently or require more from the training organization to achieve their full potential.
An interesting study commissioned by one of the most respected companies in computing found that every dollar invested in technology certification yielded more than $300 in return. If you are not already certifying your students, consider piloting a program. All the research conducted by CompTIA and others that I’ve read indicates that the result of certification is higher productivity and job satisfaction.
My purpose in writing this article was to use the case studies to take some of the mystery out of ROI calculation. I hope that readers now have a better understanding of what to measure and how to start. As learning professionals, we know the intrinsic value of education. Now we can work with our colleagues in other departments—colleagues who are already involved in ROI measurement—and use readily available data to forcefully and credibly demonstrate to the CEO and the CFO the compelling impact that training is having on the organization’s bottom line.
Gene Salois is vice president of certification programs at CompTIA, www.comptia.org. Gene has more than 23 years of industry expertise in training and certification. He previously served as director of certificaton programs and performance assessment at Lucent Technologies.
