We are back! After a brief hiatus, we continue our UGESP Series by exploring the UGESP concept of suitable alternative selection procedures addressed by the UGESP. This piece explores the difficulty in determining (a) whether two selection procedures are truly alternatives to one another and (b) what level of effort to identify a suitable alternative is considered reasonable.
Background
To the extent that a selection procedure results in adverse impact against a protected class (e.g., women, men), it is incumbent on the employer to (a) ensure that decisions based on selection procedure scores are valid and (b) evaluate the availability of alternative selection procedures that are equally valid for the intended purpose and result in lower adverse impact. As the UGESP state:
Where two or more selection procedures are available which serve the user's legitimate interest in efficient and trustworthy workmanship, and which are substantially equally valid for a given purpose, the user should use the procedure which has been demonstrated to have the lesser adverse impact… Whenever the user is shown an alternative selection procedure with evidence of less adverse impact and substantial evidence of validity for the same job in similar circumstances, the user should investigate it to determine the appropriateness of using or validating it in accord with these guidelines.
In short, if two selection procedures have similar levels of validity evidence but different levels of adverse impact, the procedure associated with lower adverse impact should be used. Thus, consideration of multiple selection procedures is an important aspect of quality validation studies. As stated in the UGESP:
…whenever a validity study is called for by these guidelines, the user should include, as a part of the validity study, an investigation of suitable alternative selection procedures and suitable alternative methods of using the selection procedure which have as little adverse impact as possible, to determine the appropriateness of using or validating them in accord with these guidelines.
Such guidance is sensible and should serve to result in the employment of selection procedures that maximize the validity of selection decisions while minimizing adverse impact against protected classes. However, two primary questions must be adequately answered to ensure that (a) selection procedures considered alternatives are truly alternatives and (b) adequate effort is devoted to identifying suitable alternatives. Those question are:
- What constitutes an alternative?
- What level of effort is reasonable?
Is Procedure B truly an alternative to Procedure A? A suitable alternative is defined as a selection procedure that is substantially equally valid to a different selection procedure. Unfortunately, the UGESP and case law do not clearly address what constitutes substantially equally valid. As addressed in an earlier post, questions of validity refer to whether the decisions made based on a set of selection procedure scores are valid. That is, validity refers to whether the conclusions made based on a set of scores are justified. Thus, it is reasonably inferred that substantially equally valid refers to whether the validity research on each of the procedures is equally robust and indicates that:
- The two procedures measure the same or very similar characteristics
- The two procedures predict the same work-related behaviors equally well
Unfortunately, the idea of suitable alternatives is often over-simplified and thought to mean that two selection procedures have similar validity coefficients. Such a perspective ignores that the two procedures may be measuring substantially different characteristics and predicting substantially different aspects of work performance. For example, cognitive ability tests are typically associated with higher racial subgroup differences than are personality inventories, but in a particular situation, the validity coefficients associated with each may be similar. If one were to consider a personality inventory a suitable alternative for a cognitive ability test, one would be ignoring that the two selection procedures predict different on-the-job behaviors. In our view, such a perspective is inconsistent both with UGESP guidance and with contemporary selection research and guidance.
Take for example a police department that is using a reading comprehension test to ensure that its newly hired officers can understand the material presented in the academy (e,g., constitutional law, agency policies and procedures). The validation study shows that scores on the reading test predict performance in the academy as well as on-the-job performance in such dimensions as report writing, court testimony, and decision making. Because the test results in high levels of adverse impact, a civil rights group suggests that the department use a personality inventory instead of the cognitive ability test. The group’s argument is that both correlate .30 with academy performance but the personality inventory has less adverse impact. Is the personality inventory a reasonable alternative? No, because the two tests are tapping completely different characteristics.
If a particular selection procedure measuring cognitive ability is proposed to be used, other selection procedures measuring cognitive ability, but with smaller subgroup differences, should be considered as suitable alternatives. For example, if a traditional academic-based test measuring verbal and mathematical reasoning is being considered for selection purposes, a suitable alternative might be a test that measures reasoning via questions that ask about patterns in sets of symbols or diagrams. Each test is intended to measure reasoning ability, and the latter may be equally valid for selection into the target job yet may have smaller subgroup differences. This assumes that verbal and mathematical reasoning ability, per se, are not the characteristics critical to job performance, but rather the broader characteristic of reasoning ability is what really matters.
Essentially, a suitable alternative to a selection procedure under consideration is one that taps the same job-relevant characteristics but taps fewer job-irrelevant characteristics that exacerbate subgroup differences. Questions that require answers in the search for suitable alternatives include:
- Is there an alternative procedure that is intended to measure the same or a highly similar characteristic(s)?
- Is there sufficient evidence that the procedure effectively measures such characteristics?
- Is there approximately equal job-related evidence for that procedure as for the initial procedure?
- Are sub-group differences in scores obtained from that procedure meaningfully smaller than those for the procedure I am considering?
What is the burden to seek and identify an alternative? The UGESP note that use of a selection procedure producing adverse impact is lawful if, “a user has made a reasonable effort to become aware of such alternative procedures and validity has been demonstrated in accord with these guidelines...” They further note that:
…preliminary determination of the suitability of the alternative selection procedure for the user and job in question may have to be made on the basis of incomplete information…the investigation should continue until the user has reasonably concluded that the alternative is not useful or not suitable, or until a study of its validity has been completed.
But what is a reasonable effort? Thankfully, the UGESP Questions and Answers provide explicit guidance concerning the limits of a user’s effort and outline a series of steps to take to meet the burden.
- “A reasonable investigation of alternatives would begin with a search of the published literature (test manuals and journal articles) to develop a list of currently available selection procedures that have in the past been found to be valid for the job in question or for similar jobs.”
- “A further review would then be required of all selection procedures at least as valid as the proposed procedure to determine if any offer the probability of lesser adverse impact."
- Note the earlier discussion concerning the definition of “equally valid.”
- “Where the information on the proposed selection procedure indicates a low degree of validity and high adverse impact, and where the published literature does not suggest a better alternative, investigation of other sources (for example, professionally-available, unpublished research studies) may also be necessary before continuing use of the proposed procedure can be justified."
- “In any event, a survey of the enforcement agencies alone does not constitute a reasonable investigation of alternatives."
- “Professional reporting of studies of validity and adverse impact is encouraged within the constraints of practicality."
The composite caveat One question that remains unanswered by UGESP and case law, to our knowledge, is whether a composite selection procedure, in which scores from different tests of different job-related characteristics are combined, presents a suitable alternative to just one test of one type of job-related characteristic. For example, it is unclear whether a composite selection procedure combining scores from a job-related cognitive ability test and a job-related personality inventory presents a suitable alternative to the cognitive ability test alone.
If the composite has stronger evidence of validity and smaller subgroup differences than the cognitive ability test alone, is the composite a suitable alternative? Perhaps it is because, unlike a situation in which a personality inventory is proposed as a suitable alternative to a cognitive ability test, the composite predicts both work behaviors associated with cognitive ability and work behaviors associated with personality.
A more complex scenario is one in which the composite has stronger evidence of validity but higher subgroup differences than the cognitive ability test alone. Such a scenario could arise if the personality test has subgroup differences, however small, in the same direction as the cognitive ability test. Because the two selection procedures are indicators of different characteristics, the subgroup differences may compound rather than average out. The question is whether the composite should be used because it has stronger validity evidence or whether the cognitive ability test should be used alone because it has smaller subgroup differences. Without clear guidance from the UGESP or case law, such a choice may come down to the values and objectives of the organization considering the selection procedures.
Of course, considerations of cost, timing, and other practical realities are relevant to whether investigations of available composites exceed the threshold of reasonable effort. Additionally, we have not addressed questions of how different components might be weighted, how to determine appropriate cut-scores, and whether different configurations of elements in a composite reflect suitable alternatives to one another. The answers to such questions are context dependent and are not readily explored in a single blog. What is clear from the UGESP, however, is that combining different selection procedures into composite measures is not prohibited by the suitable alternatives search requirement. The UGESP state:
Whenever the user is shown an alternative selection procedure with evidence of less adverse impact and substantial evidence of validity for the same job in similar circumstances, the user should investigate it to determine the appropriateness of using or validating it in accord with these guidelines. This subsection is not intended to preclude the combination of procedures into a significantly more valid procedure, if the use of such a combination has been shown to be in compliance with the guidelines” (emphasis in bold added).
Conclusion The requirement to seek and evaluate suitable alternative selection procedures can be a complex issue in practice. Unfortunately, guidance on the topic is relatively scarce, and there is widespread misinterpretation of what constitutes a suitable alternative and what efforts are required to seek out a suitable alternative. DCI’s Kayo Sady is co-editor of an upcoming handbook (to be published in 2014) that will provide concrete, practitioner-oriented guidance on a number of HR legal issues. The volume will include a thorough treatment of the suitable alternatives requirement including evaluation of the UGESP, case law, the research literature, case studies, best practice guidance, and attorney commentary.
by Kayo Sady, Ph.D, Consultant, Eric Dunleavy, Ph.D., Principal Consultant, and Mike Aamodt, Ph.D., Principal Consultant, DCI Consulting Group
