How Recruiting Software Evaluates Job Candidates

Although Jane thinks this is science fiction, software exists to perform all of these tasks. And employers are beginning to use it. You tell Jane this, and briefly describe several examples that you have read about. Resume evaluation.

Many in the federal HR community are familiar with Resumix, originally as a standalone product and in recent years as part of Hiring Gateway. Resumix uses a database of skill descriptions to match the skills in a job announcement with the same skills in an applicant’s resume.

Most users are aware that selecting the right words is the key strategy for both employer and applicant alike. Some employers worry that applicants can “fake” higher qualifications by clever use of keywords.

Others coach some or all applicants in how to use keywords to best effect. Many remain unsure if the faking, the coaching or even the resume evaluation is effective. Essay scoring. Essay scoring by computer has a long history.

Ellis Page began developing PEG (Project Essay Grade) software as an aid for high-school English teachers in the late 1960’s. In the last decade, the Educational Testing Service (ETS) and a small group of other testing organizations have used software that grades essays in educational settings.

While acknowledging that its “e-rater” software is not as “intelligent” as a human being, ETS has shown that e-rater can classify written essays into scoring categories as well as teams of trained raters.

Government personnel psychologists and federal contractors such as BrainBench, are extending this technology to score written job-application materials. Training-needs analysis. Text mining, also known as data mining or knowledge discovery, uses database searches and pattern-recognition techniques to find useful information in large collections of documents.

Knowledge Analysis Technologies, also an early developer of essay scoring software, has used text mining to improve the efficiency of training for the U.S. Air Force. Its CareerMap software examines training materials and training and experience descriptions written by Air Force personnel.

The system identifies individuals who know “almost everything” they need to accomplish a particular mission. CareerMap then identifies training materials needed to close the gap between “almost” and “everything” to prepare the selected personnel to perform. Text analysis is appearing in other HR processes as well.

For example, SPSS, a company that produces statistical analysis software and data mining software, has recently released “SPSS Text Analysis for Surveys,” designed for analysis of open-ended questions in Web-based surveys. HR specialists who survey their employees will be using such tools in the near future.

What Is This? Jane takes another large bite of her sandwich, impressed with what you know about these programs. She wonders what they have in common. You cannot understand the specifics of every text-analysis program.

Marketing brochures usually emphasize the uniqueness of each product, but there are some common elements to all programs. You can find out what basic technology each product uses and use this information to ask good questions to find out more. Basic vocabulary. All text analysis programs find patterns in text differently, but all begin with individual words and phrases. They look for specific words, or lists of words, and count how often they find them.

When you are discussing text-analysis software with a vendor or programmer, ask how the system picks out specific words and what it does with them afterward. Software knowledge. Each text analysis program brings to bear some kind of “knowledge.”

Some programs contain a large number of “rules” that work together to reach conclusions about word meaning (IF ‘audit’ IS NEAR ‘account’ OR ‘spreadsheet’ THEN ADD ‘Accounting’ TO SKILLSET). Others determine which words appear together in text documents and use statistical learning algorithms to summarize these relationships.

The “knowledge” in these systems is a set of statistical weights that capture how well word patterns in one document, such as a resume, match with another document, such as a job announcement. Ask vendors what kind of knowledge their systems contain.

Human knowledge. Each text analysis program is based on knowledge that comes from a human being. Some use very general human knowledge similar to dictionary definitions. Others are based on interviews with hiring officials, HR specialists or others with expert knowledge about the language used in HR documents.

And some analyze documents written by experts for language patterns. Always ask a system developer how their text analysis system is grounded in human knowledge. Ask text-analysis software developers questions about the vocabulary their system recognizes, the knowledge it contains and how that knowledge was obtained from experts.

How Good Is It? Jane has finished her sandwich, and has also absorbed your overview of this technology. Now she just wants to know, how good is it? Software couldn’t possibly be as good as a human being, could it? It can be difficult to evaluate. Many programs are commercial and proprietary; developers are not motivated to describe them in detail.

The technology itself presents a moving target with change driven by innovations in software design, statistical-language learning, and other disciplines. You resist the easy alternatives — either rejecting this technology because it is new and outside your experience or embracing it because it is, well, new and outside of your experience. Remembering a few things will help you to evaluate these programs. Reliable. These programs are relentlessly reliable.

Unlike people, they do not tire, lose focus or slip out for an early lunch. Just like a spell checker will always find “teh” instead of “the,” a resume program will always find “exceptional spelling skills” no matter what part of your resume it is in. This should not impress us, though. The issue is whether a program does the right thing, not that it can do the wrong thing the same way every time.

A test is a test. Remember that these programs are subject to the uniform guidelines and other standards that govern personnel practices. Advanced technology may seem like magic, but the magician has to explain the tricks if they are used to hire. Reminding software vendors of this fact can persuade them to share more details about how a particular system works. No experience. These programs do not exercise judgment or apply work experience to unanticipated situations. And they will not determine whether an applicant really has the skills claimed on a resume.

They will find both “teh” and “exceptional spelling skills” in your resume. But they will draw no conclusions from finding either unless this knowledge was explicitly designed into them. (They won’t fall on the floor laughing, either.) The right standard. No program is as “intelligent” as a person is, so this is not a useful comparison.

Instead, consider what happens when several people evaluate a resume. The hiring decision is not based on the knowledge or judgment of any single individual. It is based on what they agree on, while using a well-defined set of criteria. In a sense, the team acts as a kind of “technology” in making a reliable decision that is consistent with the written guidelines. This is the appropriate standard of comparison — can a text analysis program match the decisions people make? Look for the evidence.

Educational researchers, psychologists, and others have conducted studies to determine whether these programs can match the decision-making behavior of trained human beings. Some of them can. Instead of asking general questions about how “smart” text-analysis software is, ask if there is any evidence that the software can classify essays, resumes or other target documents as well as people can. Look for a formal report of this comparison. What Can You Learn? Unsatisfied with both her sandwich and your answers, Jane scans the dessert menu. She thinks she needs to know more about this technology. You think you do, too.

Here are some things you and Jane can do. Try a Web demo. Two essay scoring programs have Web sites that allow you to see how they score sample essays. They include Knowledge Analysis Technologies and Project Essay Grade. You can use your knowledge of job applicants to investigate how these systems work. Choose a sample essay and see how it is scored after you introduce various errors you have seen applicants make.

You might try some great writing, too, and see if they give you credit for it. Try a simple tool. An extensive, vendor-neutral list of text-analysis software is available on the Web at Some of these programs are commercial and complex. But some are freeware and can be learned quickly. Download one and become familiar with it. This will give you a feel for more sophisticated applications in the same way that taking so many tests in college gave you a basic understanding of more complex selection and promotion processes. You may decide to become the office (or agency) expert.

Use content you know. If you try out these programs using documents that you already know well, it will be easier to discover their strengths and limitations. You know your own resume better than anyone — what can an automated text analysis tool discover about it? Less than you know, of course, but what kinds of things does it miss?

Are there patterns you have not noticed? An article from shared the 100 most common keywords used by employers when searching resumes. Try using this list, or a list of skill terms from a recent job announcement. At the very least you will learn to ask better questions about the way this technology is used. Join the discussion. You do not need to be an expert to talk with others about this technology. Three e-mail lists can help you learn more through discussion.

The Content Analysis News and Discussion list encourages discussion among researchers who analyze documents. Computer-based text analysis is often discussed and the group provides friendly and informative responses to questions from newcomers. KDNuggets takes a more applied stance toward “knowledge discovery” in business documents and databases. More tech-savvy readers may prefer to discuss programming issues with the Information Retrieval list. The FAQs, archives, and helpful participants of these lists can all help you learn more about text-analysis technology.