“In hindsight, we probably should have …” is the opening line for many discussions about a function or organizational unit that has gone wrong. It is easy to say a well-designed, balanced performance measurement system that drives appropriate stakeholder behaviors greatly reduces the likelihood of function failure. Further, it is reasonably easy to say, in the absence of a well-employed performance measurement system, success is achieved only by chance rather than design. It is more difficult to actually design and execute a good performance measurement system.
To establish product and service norms and the function’s image, it is important that managers give objective meaning to the elements of the service experience, the product and the expected image result. Only clear definitions can lead to meaningful measures of the important characteristics dictated by the organization’s unique needs. The measurement system often will dictate how the scales are tipped in the balancing of costs and the benefits derived from the expenditures made. Performance measures can nudge employees to optimize the relationships of costs and outputs, or they can encourage suboptimization either through cutting corners or wasted effort.
In the quest to integrate the value of services to the client and the function’s objectives, the measurement challenge simultaneously is to recognize stakeholder concerns and create credible measures that balance management and other stakeholder needs. Finally, in the creation of client loyalty, all the variables that lead to client satisfaction come into play. The family of measures created needs to adequately sample the most critical attributes of the service.
Measurement in Support of the Training Function
Too often in companies in which the training function is viewed only as a cost center, providing some tenuous benefits to the individual or the company, the function’s complexity is not understood, and its potential goes unrealized. Organizations that link the training function’s performance to company performance are better positioned to realize its benefits. The first job in showing training’s value is the thoughtful and systematic measurement and evaluation of the function’s performance.
Specific circumstances dictate the importance and value of each measure, and caution should be taken to not emphasize one important measure to the exclusion of others. An out-of-balance measurement system emphasizes one set of critical stakeholder concerns while neglecting others. Consider the training function that is so concerned with on-time, on-budget production that it cuts corners to get the product out the door and suffers cycles of rework and a poor reputation, or the manager who overinvests in marketing, generating demand without having done adequate planning to deliver on the promise. Having a balanced set of measures is a necessary condition for a good performance measurement system.
At the same time, it is important to have a few good measures that fit the circumstances rather than have a long list of measures, which tends to overwhelm stakeholders and, thus, drives no related action. Proper measurement involves the careful collection of representative data used to make informed decisions relative to the value, effectiveness and modification of the training function. The measurement system enhances the performance of the function by driving the appropriate behavior of its stakeholders.
As noted above, adequate measurement of the training function begins with an understanding of the business variables that affect its overall performance. These variables represent specific stakeholder concerns and categories of functional performance, and in more practical terms, they are the areas critical to success that the training function’s leadership should monitor.
The Business and Management Variables of Most Concern
Identifying the areas for measurement begins with recognizing, in detail, the stakeholders of the training function and their expectations. Figure 1 presents the training function’s main stakeholders and some key issues for which the training function might need good measures in order to respond.
With the stakeholders’ wants clearly in mind, attention can be turned to identifying the critical areas of measurement. Each training function will need a somewhat different emphasis on measures because each operates in a different context of technology, organizational structure, political forces and other influences. Still, the measures of importance for training functions always focus on inputs, processes, outputs and outcomes, and the common key measurement areas generally include at least the following four performance categories:
1. Finance: The use and flow of money within the function is critical to its health. Insufficient or poorly used funds might jeopardize the function.
2. Products and Services: The quality and usability of output is key to meeting client expectations and growing the business.
3. Human Capital: The human capital measures for the function focus on quality, quantity and availability of personnel.
4. Infrastructure: The training function must have access to the core structural and functional elements required for the day-to-day training business.
These categories of performance are important to all the stakeholders of the training function. Specific circumstances dictate the exact measures needed to service each one, but it is always important to balance the measures to drive appropriate behaviors across all stakeholders.
The Application of Measures to the Variables of Concern
Understanding the vital components of each category is fundamental to the measurement-balancing act and to the composition of appropriate measures. Decomposing the category into a manageable set of components requires taking an analytical look at the category as it is conceived and used in the company. A hypothetical decomposition for the four performance categories noted above is presented in Figure 2. The figure provides a small sampling of potential vital components for each area of focus and performance category, as well as a sample of lag and lead indicators for the vital components. The candidate measures for each component should include both key lagging and leading indicators of performance. Lag indicators are important due to their evaluative value, and lead indicators are measures of a variable in today’s environment that signal changes in other variables in the future. Through clear comprehension of these subparts and their relationship to the unique context of the company, leadership can balance the efforts to drive the desired behaviors in the training function based on specific intent.
Whereas the categories are consistent throughout most training functions, the specific components of each — leadership’s areas of focus — will vary. Moreover, the areas of focus and the resulting most relevant measures will change, as dictated by dynamic company and market forces, shifts in strategy and competitive influences.
Ultimately, training management must define the pattern of the vital few components that describe the overall success path for the function in a time frame during which company strategy is generally stable but whose beginning or end might be marked by key events (e.g. a major change in the regulatory environment or a change in the company’s dominant coalition of leaders). Once the pattern of the vital few components for the function is identified, the hard work begins: the work of identifying valid and reliable measures that will drive the right stakeholder behaviors in regard to the function.
Selecting the Best Set of Measures
With the vital components identified, specifically relevant measures for each component can be established. The process of identifying and implementing measures is widely practiced. What is rarely understood, however, is that each measure’s value is predicated on its fulfillment of two purposes.
The first purpose is well-known and relatively straightforward: The measures are to provide an indication of changes in real performance in the areas they measure. This is the construct validity of the measure. The second purpose, while equally important, is frequently overlooked. The measures themselves directly influence the behavior of those charged with producing the measurables. Simply put, measurement drives behavior. This is referred to as consequential validity. Therefore, any one measure must be viewed on the construct merits of the data and on the consequential behavior it produces.
The decision to take these measures — or a host of similar ones that might be applicable — is dictated by what, in the function’s specific context, best represents the component being measured. Often, these measures are “on their face” direct measures of the vital component. They provide valid assessment of the construct, in this case, “on-time performance.” Although necessary, this kind of validity alone is not enough.
The potential behaviors driven by the measures represent both desirable and undesirable consequences for the organization. Measures that produce positive behaviors naturally would be desirable in the performance measurement system. But the fact is almost any unchecked measure will drive undesirable behavior.
Therefore, there is a clear need for a balanced family of measures to mitigate the negative effects of any one measure. Equally emphasizing a measure of quality (a lead indicator for client satisfaction) would guide an employee to meet project deadlines (a lag indicator for stakeholder satisfaction) while maintaining the quality of the product.
Similarly, a financial measure regarding cost management (e.g. actual to planned spend — a lag indicator for management effectiveness) will tend to balance the “increased cost behavior,” serving to raise questions about prioritization and optimizing the major project elements of scope, resources, timelines and quality. The aggregate effect of a family of measures will shape stakeholders’ behaviors according to the pattern, emphasis and design of that family of measures. Doing the behavioral or consequential analysis is a key ingredient to building a professional, effective performance management system for the training function.
Tying It Together
In the determination of the measurables, it is critical to evaluate each on the information it provides and the behaviors it might generate. Establishing such measures in the context of multiple stakeholder expectations and a dynamic environment is a complex task. It is essential that the architect of a performance management system know the theory of the function, its business model.
Also, two of the most critical messages about building a good performance measurement system are to identify the vital few components that are unique success factors for your organization in a time period and to apply balanced measures with a full analysis of the stakeholder behaviors that are likely to emerge as a consequence of the particular measures used.
A process that brings together these key requirements is necessary to get to a good, custom set of lagging (essentially evaluative) and leading (essentially predictive) performance measures. Such a process produces management foresight, an outcome that, if achieved, greatly reduces the risk that some executive will someday say of your function, “In hindsight, perhaps we should have …”
Oliver W. Cummings is a manager of custom development programs at Riverside Publishing. He can be reached at editor@clomedia.com.
