Chief learning officers often must evaluate their key learning programs, collecting several types of data—reaction, learning, application, impact, intangibles and maybe even return on investments. What if the evaluation produces disappointing results? Suppose application and impact were less than desired, and the ROI calculation negative. This prospect causes some learning executives to steer clear of this level of accountability altogether.
For some CLOs, negative results are the ultimate fear. Immediately, they begin to think, “Will this reflect unfavorably on me? On the program? On the function? Will budgets disappear? Will support diminish?” These are all legitimate questions, but most of these fears are unfounded. In fact, negative results reveal the potential to improve programs. Here are 11 ways to address negative results and use them to facilitate positive transformations:
1. Recognize the Power of a Negative Study
When the study results are negative, there is always an abundance of data indicating what went wrong. Was it an adverse reaction? Was there a lack of learning? Was there a failure to implement or apply what was learned? Did major barriers prevent success? Or was there a misalignment in the beginning? These are legitimate questions about lack of success, and the answers are always obtained in a comprehensive evaluation study.
2. Look for Red Flags
Indications of problems often pop up in the first stages of initiation—after reaction and learning data have been collected. Many signals can provide insight into the program’s success or lack of success, such as participants perceiving that the program is not relevant to their jobs. Perhaps they would not recommend it to others or do not intend to use it on the job. These responses can indicate a lack of utilization, which usually translates into negative results. Connecting this information requires analyzing data beyond overall satisfaction with the program, the instructor and the learning environment. While important, these types of ratings may not reveal the value of the content and its potential use. Also, if an evaluation study is conducted on a program as it is being implemented, low ratings for reaction and learning may signal the need for adjustments before any additional evaluation is conducted.
3. Lower Outcome Expectations
When there is a signal that the study may be negative, or it appears that there could be a danger of less-than-desired success, the expectations of the outcome should be lowered. The “under-promise and over-deliver” approach is best applied here. Containing your enthusiasm for the results early in the process is important. This is not to suggest that a gloom-and-doom approach throughout the study is appropriate, but that expectations should be managed and kept on the low side.
4. Look for Data Everywhere
Evaluators are challenged to uncover all the data connected to the program—both positive and negative. To that end, it is critical to look everywhere for data that shows value (or the lack of it). This thorough approach will ensure that nothing is left undiscovered—the fear harbored by many individuals when facing negative results.
5. Never Alter the Standards
When the results are less than desired, it is tempting to lower the standards—to change the assumptions about collecting, processing, analyzing and reporting the data. This is not a time to change the standards. Changing the standards to make the data more positive renders the study virtually worthless. Without standards, there is no credibility.
6. Remain Objective Throughout
Ideally, the evaluator should be completely objective or independent of the program. This objectivity provides an arms-length evaluation of its success. It is important not only to enter the project from an objective standpoint, but also to remain objective throughout the process. Never become an advocate for or against it. This helps alleviate the concern that the results may be biased.
7. Prepare the Team for the Bad News
As red flags pop up and expectations are lowered, it appears that a less-than-desired outcome will be realized. It is best to prepare the team for this bad news early in the process. Part of the preparation is to make sure that they don’t reveal or discuss the outcome of the program with others. Even when early results are positive, it is best to keep the data confidential until all are collected. Also, when it appears that the results are going to be negative, an early meeting will help develop a strategy to deal with the outcome. This preparation may address how the data will be communicated, the actions needed to improve the program and, of course, explanations as to what caused the lack of success.
8. Consider Different Scenarios
Standards connected with the ROI methodology are conservative for a reason: The conservative approach adds credibility. Consequently, there is a buy-in of the data and the results. However, sometimes it may be helpful to examine what the result might be if the conservative standards were not used. Other scenarios may actually show positive results. In this case, the standards are not changed, but the presentation shows how different the data would be if other assumptions were made. This approach allows the audience to see how conservative the standards are. For example, on the cost side, including all costs sometimes drives the project to a negative ROI. If other assumptions could be made about the costs, the value could be changed and a different ROI calculation might be made. On the benefit side, lack of data from a particular group sometimes drives a study into negative territory because of the “no data, no improvement” standard. However, another assumption could be made about the missing data to calculate an alternative ROI. It is important for these other scenarios to be offered to educate the audience about the value of what is obtained and to underscore the conservative approach. It should be clear that the standards are not changed and that the comparisons with other studies would be based on the standards in the original calculation.
9. Find Out What Went Wrong
With disappointing results, the first question usually asked is, “What went wrong?” It is important to uncover the reasons for the lack of success. As the process unfolds, there is often an abundance of data to indicate what went wrong. The follow-up evaluation will contain specific questions about impediments and inhibitors. In addition, asking for suggestions for improvements often underscores how things could be changed to make a difference. Even when collecting enablers and enhancers, there may be clues as to what could be changed to make it much better. In most situations, there is little doubt as to what went wrong and what can be changed. In worst-case scenarios, if the program cannot be modified or enhanced to add value, it may mean that it should be discontinued.
10. Adjust the Story Line
When communicating data, negative results indicate that the story line needs to change. Instead of saying, “Let’s celebrate—we’ve got great results for this program,” the story reads, “Now we have data that show how to make this program more successful.” The audience must understand that the lack of success may have existed previously, but no data were available to know what needed to be changed. Now, the data exist. In an odd sort of way, this becomes a positive spin on less-than-positive data.
11. Drive Improvement
Evaluation data are virtually useless unless used to improve processes. In a negative study, there are usually many items that could be changed to make it more successful. It is important that a commitment is secured to make needed adjustments so that the program will be successful in the future. Until those actions are approved and implemented, the work is not complete. In worst-case scenarios, if the program cannot be changed to add value, it should be terminated and the important lessons should be communicated to others. This last step underscores that the comprehensive evaluation is used for process improvement and not for performance evaluation of the staff.
Negative study results do not have to be bad news. Negative results contain data that can be used not only to explain what happened, but also to adapt and improve in the future. It is important to consider the potential of a negative study and adjust expectations and strategies throughout the process to keep the negative results from being a surprise. In the worst-case situation, negative data will surprise the key sponsor at the time of presentation.
Jack J. Phillips, Ph.D., chairman of the ROI Institute, developed and pioneered the ROI process and has written more than 15 books on the subject. Jack can be reached at jphillips@clomedia.com.
