Applicant questionnaires and gaming the system

This column was originally published on Jeff Neal’s blog, ChiefHRO.com, and was republished here with permission from the author.

Only 36 percent of employees who responded to the 2017 Federal Employee Viewpoint Survey said promotions in their work unit are based on merit. There are many reasons for the lack of employee and hiring manager confidence in promotion programs. Concerns about “preselection” and favoritism are high on the employee list. Those reasons assume that managers and HR folks are deliberately gaming the system to get to favored applicants.

Maybe those reasons drive some of the problems with merit promotion and competitive hiring programs, but they are not the only problems. In fact, I believe they are not the biggest problem. That honor goes to the assessment process most agencies use for their job announcements. Anyone who has applied for a job in recent years has come across an applicant questionnaire. Let’s just call it an AQ for short.

The AQ is exactly what it sounds like — a lot of questions that purport to distinguish the best candidates from the rest of the pack. In theory, they are a great idea. Simple questions that tell a manager who is among the best qualified (and who is not), without putting a large burden on applicants, should make the hiring process work better. Except when they don’t.

Before we get too far into the AQ, let’s get in the wayback machine and go back in time, to the era before we had the AQ. Then we had KSAs. Remember those? KSAs are knowledge, skills and abilities, and job applicants had to write lengthy narratives describing how their education, training and experience matched the KSAs for the job. Virtually no one liked KSA narratives. They were time-consuming to write, miserable to read, and many folks believed they encouraged applicants to lie or to embellish their qualifications. The AQ was intended to eliminate those problems and improve the hiring process for HR, applicants and hiring managers. It is safe to say that it did not work out quite like it was expected.

What happened? Why did what seemed like a great idea not make the kind of difference everyone expected? I believe there are three reasons.

Hiring managers are not involved in the process as much as they should be. A good AQ requires input from someone who has in-depth understanding of the job being filled. That means an experienced supervisor or subject-matter expert (SME). Too often they are done by HR alone. Even the best HR specialists are not SMEs. The Office of Personnel Management (OPM) recognized the problem when it created its hiring excellence program, which is intended to bring HR and hiring managers together to fill jobs. Unless agencies expand their use of SMEs, the problem with poor quality AQs will not go away.

People lie. This one should not be a surprise. The people who lied when they wrote KSA narratives are still lying. The difference is that it is easier and faster now. No need to write 500 words to inflate your experience — just check the box and go on to the next question. Sadly, the number of people who lie on job applications seems to be going up. When we started using the AQ (about 25 years ago), lying in response to the questions did not appear to be a big problem. Now it is.

Many of the AQs are just terrible. Even if we solved the problems with lying and lack of SME involvement in the process, the AQs we see for many jobs have to be better. There are two big problems with them. The first is that they take a basic format — five answers ranging from “I don’t know anything” to “I am an expert” and use them for every question. The second is that there are far too many questions. Here is an example of the first problem (from a current job announcement):

Finding an example of this type of AQ took 30 seconds. They are everywhere. From the perspective of simplicity, they seem great. Come up with a job requirement, then drop in these responses. The first three answers mean you have never done the work, or you have only done it with close supervision. You have to get to answer #4 to find someone who actually has decent experience. Even #4 does not mean you did it well. It did not take long for a people to figure out that selecting answer 5 would get the most points. It is also hard to prove an applicant who selects answer #5 lied about it. “Considered an expert” is vague, as is “supervised performance.” Almost anyone can say they are “consulted by other workers.” The fact that the hiring and promotion processes are not highly regarded adds to the problem, making it easier for people who would never consider themselves to be liars to pick answer #5, even if they are not objectively what anyone would call an expert. People have told me that they pick the top answer because “It is a way to get to the interview process.” With this type of pattern, we should not be surprised when hiring managers complain that they get lists of people who cannot do the job.

The second part of the problem with AQs is the number of questions. It is common to see 20 or more questions per job. Many of those are junk questions that do not differentiate the great applicants from the average ones. If there are five questions that address the core requirements of the job, and 15 questions that are more general, such as “ability to communicate orally,” it is likely that people who have the necessary skills will not make the cut.

If we want to make the hiring process better, and we want to build employee trust in promotion programs, we have to stop misusing the AQ. They must be tailored to the job and they must have something other than the same five answers. They need to differentiate between good candidates and bad candidates in a way that the five standard responses cannot. And finally, we have to recognize that the AQ is only one part of the evaluation process. When OPM got rid of KSAs, they told agencies they could not use them in the initial screening process. They also said agencies could use multiple assessments. What happened was only the first part. We got rid of the KSAs, but we did not use additional assessments. We may not see comprehensive hiring reform any time soon, but we can and must improve the assessment process now.

Jeff Neal is a senior vice president for ICF and founder of the blog, ChiefHRO.com. Before coming to ICF, Neal was the chief human capital officer at the Homeland Security Department and the chief human resources officer at the Defense Logistics Agency.

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.