Unless you boycotted popular media, even non-sports-fans couldn’t avoid hearing that the Boston Red Sox won their first World Series in 86 years this past fall. Their decades-long failure was largely attributed to the “Curse of the Bambino,” which many believe began in 1919 when the Red Sox sold Babe Ruth (aka “The Bambino”) to their hated rivals, the New York Yankees. Since then, the Yankees had won 26 World Series to the Red Sox’ none. With all of the history and passion surrounding the final games of the season, it’s easy to see why the 2004 World Series received so much attention. Yet, underlying the emotion and media hysteria is a hidden story about the largely analytical way the Red Sox accomplished this long-awaited goal. This story and their methods hold several correlations for learning professionals in all organizations.
Recognizing What’s Important
Baseball is rooted in history and dominated by tradition. The majority of coaches, managers, scouts and front-office staff have largely used the same methods to evaluate personnel: trusting the mind of experience. The “experts” often ultimately relied on gut instinct and what they saw with their own two eyes to guide personnel decisions. In most situations, raw talent was the overriding factor in decisions. However, baseball is a game of numbers, and these same individuals have relied on statistics to augment their opinions.
Everything in the game today is measured, but traditionally only a subset of this data garnered much attention. The age-old, simple metrics of batting average, runs batted in and home runs have been used as the primary measurement of a hitter by almost everyone associated with the game since the late 1800s.
The problem with these common measurements is that, like many things in life, they don’t take into account the realities of the game. Batting average (hits divided by at bats), for example, is completely unaffected when a player achieves a base on balls (walks). Walking is immensely valuable to an offense because it puts a runner on base and increases the team’s chance of scoring a run—but the batting average measurement ignores it completely.
Whether it’s performance data for an individual or measurements on programs, corporate America also has largely relied on a core set of simple metrics. Progressive learning practitioners today recognize that several of these metrics are outdated, and a much more detailed analysis is necessary to make informed decisions. More importantly, there’s been increasing recognition that learning metrics are most useful when directly tied to business productivity.
Several decades ago, a small number of baseball’s progressive thinkers recognized a need for change. Ultimately, runs win games, not hits. So a new way of looking at baseball’s statistics was born.
Coined “Sabermetrics,” after the Society for American Baseball Research (SABR), this new method of statistical analysis is about the search for objective knowledge about baseball. While the world of exotic statistics has existed for at least 25 years, it didn’t become widely popular until the best-selling book “Moneyball: The Art of Winning an Unfair Game” was authored by Michael Lewis in 2003. In the book, Lewis followed highly regarded Oakland A’s General Manager Billy Beane for an entire season, showing how he used unorthodox thinking to beat his competitors despite having one of the smallest payrolls in the game. Since Beane took over as general manager in 1997, the A’s have been one of baseball’s most successful teams by selecting productive players who are often overlooked by other teams. Sometimes short, overweight or with unusual playing styles, these players didn’t always look like the prototypical major-leaguer. Mainly, Beane placed a premium on production, not talent. His team’s consistent winning records were a testament to his beliefs, and “Moneyball” credited Beane’s affinity for Sabermetrics as the reason the A’s were so successful.
Another general manager who subscribed to the idea of Sabermetrics was Theo Epstein, the current Red Sox GM and, at 29 years old, the youngest GM in the majors. Despite his youth, Epstein did what no other Red Sox GM since 1918 has done: He won the World Series, the first “Sabermetrician” in a GM role to do so. It was Epstein’s belief in non-traditional, detailed data that helped shape the team he started the season with, and that helped him develop the team as the season progressed.
The success of the Red Sox and their performance evaluation methods call to mind some interesting links with the way today’s leading organizations are using unorthodox measures to achieve their own brand of success.
The New Measures of Performance Management
Sabermetrics is concerned both with determining the value of a player versus his peers at present, and with trying to predict the future value of a player based on his past performance. While snapshots of a player’s statistics are important, it’s the trends in these metrics that often tell the story of an up-and-coming star, or a player fading past his prime.
Clearly, corporations have been using this concept in different areas of personnel assessment for years. Behavioral-based interviewing, for example, models past performance as the biggest indicator of future performance and provides techniques with which to uncover that past performance. But this data is largely based on subjective interpretations of a candidate’s response to questions (similar to the old baseball scout evaluating talent with his own two eyes), not detailed data that has been gathered.
This is where technology plays a significant role. The Red Sox is one of many teams that now use specialized software to assist in scouting, uncovering strengths and weaknesses of opposing teams and players and deciding how much the team should be willing to pay a particular player. In fact, most followers of baseball believe that innovative uses of software have significantly changed the game.
Similarly, corporations are employing technology in new and different ways to assess personnel in their organizations. Learning management systems, which capture skill and competency data, are often used as a “qualified employee finder,” leveraging data against available job openings. Using this data can be as simple as locating an employee who knows a particular foreign language or as complicated as a sophisticated succession planning process. By assessing individuals’ proficiencies, measuring skill gaps and recording detailed information on employees’ performance, organizations are building detailed transcripts for their most important asset.
A Baseball Card for Corporations?
Like the common baseball card, which highlights each major league player, it’s not too far-fetched to envision companies possessing a snapshot of each worker, with vital performance information listed on the back. In addition to learning or HR-related information, the corporate baseball card also would likely include traditional criteria captured in various business systems. For a salesperson, tapping the company’s sales-force automation system for information such as length of time to close a sale, size of average deal and number of deals closed per month are common metrics.
These metrics, when coupled with certifications and other learning achievements, would be a powerful mini-transcript for each employee. While not in a baseball-card format, this data does exist today in reports that many organizations are producing via their learning management systems.
While objectively measuring the performance of the individual is a goal of many learning professionals, so is measuring the success of an entire program. But like traditional baseball metrics, statistical analysis of learning and performance management programs has been unscientific and rudimentary. Also like baseball, this is slowly changing. Most organizations recognize that items like courses delivered, hours trained and participant evaluation sheets can be inefficient measurements. Today’s progressive organizations are turning toward more sophisticated methods to measure the effects that learning has on performance. Several are showing impressive—albeit, non-traditional—results. Some examples include:
- Nike, the world’s leading designer, marketer and distributor of athletic footwear, apparel and accessories, has seen at least a 4 percent increase in dollars in sell-through at retail locations that have rolled out its new learning program versus those that have not. (See “In Practice” sidebar on page 24.)
- Harley-Davidson, the only major U.S.-based motorcycle manufacturer, has seen dealer satisfaction increase by 30 percent, and one product’s revenue increase 42 percent, after rolling out e-learning through Harley-Davidson University Online.
- Wyeth, one of the world’s largest research-driven pharmaceutical and health-care products companies, saw market share for one product rise from 0.5 percent to 5 percent among high-prescribing physicians due to a focused sales training program—an increase of 900 percent.
- The Institute for Motor Industry (IMI), the professional body for individuals working in all sectors of the U.K. automotive industry, attributes a 70 percent increase in annual revenue and a 40 percent increase in market share to instituting an online assessment tool.
- Fidelity Investments, the largest mutual fund company in the United States and the nation’s number-one provider of 401(k) retirement savings, has increased customers’ comfort level in investing by more than 50 percent, and has seen newly eligible participants two-and-a-half times more likely to enroll in their company retirement plan as a result of putting 401(k) education information online.
While these kinds of metrics are often discussed, linking learning to business performance unfortunately has been more talk than reality. According to the latest ASTD State of the Industry Report, companies are spending anywhere from 2 percent to 4 percent of payroll on training. That represents a significant commitment of billions of dollars for corporate education each year. Yet ASTD also has calculated that only 8 percent of companies try to measure the business impact of this investment.
Most baseball teams today don’t have that luxury. “We are fiscally responsible because the alternative would be a disaster,” Epstein recently said about the Red Sox. “Fiscal irresponsibility is the single quickest way to hamstring a franchise for a decade.” Increasingly, fiscal irresponsibility is also the quickest path to disaster for learning programs and the professionals who run them.
One common view is that companies do not measure learning’s relationship to business impact more rigorously because they do not have the experience, tools and infrastructure necessary to do so. It is not simply a lack of interest or importance—but lack of tools and expertise.
The Emergence of Learning Dashboards
Learning dashboards (or learning analytics) are the bridge for many organizations between measuring training for training’s sake and measuring it to see the impact on what matters most to executives: increasing revenue, decreasing expenses and improving cycle time.
Learning dashboards provide users with an interactive, real-time visual display correlating training activities with business performance data. This data is usually linked to an organization’s customer relationship management (CRM) system, its human resource information system (HRIS) or enterprise resource planning (ERP) platform. By integrating with these enterprise-wide systems, learning professionals can access key metrics to help make an impact in a particular division or function.
While this kind of linking has sometimes occurred in the past, it most often happens after the fact, in lengthy analysis projects laden with surveys and many hours of manual labor. Baseball figured out some time ago that “during the season” and “during the game” data are critical to improving performance on the field, as opposed to relying on only the post-season snapshots. Similarly, learning dashboards allow organizational performance analysis to happen in real-time and as an ongoing activity, utilizing existing systems as opposed to expensive people.
For many organizations, the tools to get started already exist. Learning management systems generally contain the data that form the basis of the analysis. Most analytics tools are designed to attach to an LMS and import data from the LMS database, as well as from other systems. The analytics database, which is often referred to as a “data mart” or “multidimensional database,” is the repository where this data gets aggregated. Once aggregated, the analytics engine provides a correlation between the two sets of information. This correlation happens in the form of charting, which provides the visual element of the dashboard.
Like their counterparts in baseball, the leaders of today’s learning organizations are exploring this correlation in-depth and demanding that positive correlations become part of any proposed learning program.
A Natural Evolution
Just as baseball’s new breed of executives needed alternative methods to outsmart the competition and buck the status quo, the growth of the senior learning professional is a major reason for the growth in popularity of new and detailed performance data. Today’s CLO is much more aware of bottom-line issues in the organization and more concerned about operational and financial factors. More progressive learning executives are taking a business approach and an analytical look at learning and development, with non-traditional data as part of the strategy.
While statistics play an important role, they aren’t everything. In business and baseball, there are many intangibles that are difficult to measure. But the better prepared a manager or a coach is with data, the less he or she has to rely on gut instinct. With the right data and the right tools, a mistake of Bambino-like proportions might even be avoided.
Kevin Oakes is president of SumTotal Systems and a lifelong Red Sox fan. Kevin witnessed history by attending Game 7 of the ALCS at Yankee Stadium and, despite paying an obscene amount of money for his ticket, he reports it was worth every penny. Kevin can be reached at koakes@clomedia.com.
