I work in the area of Natural Language Processing. I wish I had kept track of how many recruiters have contacted me in the past year, because that number would be useful here to illustrate the present state of the market.

This might sound like a nice problem to have, but many employers have only a general idea of what NLP can do for them. Many employers are hoping to find a project lead who can walk in and work miracles. NLP is seen as magic.

The demand for NLP specialists has grown enormously in the past few years. The current supply of specialists comes nowhere near meeting that demand. I see this from the number of recruiters who contact me. I also see it from the other side of the desk; when I’ve been involved in hiring others who can do NLP, we’ve had a very difficult time finding suitable candidates, and our reqs usually stay open for a long time. My offhand guess is that may take up to 10 years for the colleges and universities to crank out enough NLP specialists to backfill this market demand.

Some recruiters have told me that they’re new to recruiting for NLP. Some have been candid that they’re not entirely sure what NLP is. Some have asked for advice on finding people. More than one has said, “I can’t even find anybody.”

I’ve generally been willing to share my time and advice with recruiters so that they’re in a better position to help the industry along. However, I’ve just accepted a new position, so I won’t be spending as much time talking to recruiters. Based on many past conversations, here’s what I think recruiters need to know about NLP.





So what is NLP, anyway?

In short, NLP is any use of computers to process human language. It’s a very broad term covering a lot of technologies and applications. Here are some familiar technologies which use NLP techniques:

Voice command systems such as Siri or Alexa/Echo. It also includes voice activated telephone-based call center systems, a.k.a. IVR systems (“Tell me in a few words why you’re calling today”).

Spelling and grammar correction in a word processor such as Microsoft Word. It also includes the autocorrect feature when you text on your smart phone.

Spam filters, i.e. classifying a text as spam or not-spam. Other kinds of text classification can include classifying a movie review as positive or negative, for example.

Automated translation from one language to another (French to English, etc.). Google Translate is a well-known implementation.

This is just a brief sampling of the uses of NLP. The field is broad.





What most employers want

Despite the breadth of the field, my experience is that most recruiters are looking to fill positions for one particular type of NLP work. Usually, the employer has a ton of text data—let’s take patient medical notes as a typical example—and the employer knows that it would be very useful to extract information from it. It is very expensive to pay trained humans to read the text, to make informed decisions about what it means, and to key the desired information into some machine-readable form. From a standpoint of cost and speed, wouldn’t it be great if we could replace this fleet of humans with a rack of servers?

This type of NLP is called “semantic extraction”, or pulling the meaning out of text. It goes without saying that there are all sorts of business opportunities in this area. Unfortunately, the technical problem is not an easy one at all.

I’ve been working in NLP for a good while; but when an employer wants me to be the tech lead on a semantic extraction project, I’ve been turning these positions down. I don’t think I’m ready to lead that kind of project, although maybe I will be in a few years. I don’t want to take a job of this kind unless I’m fairly confident that I can deliver on the high expectations which these employers are setting, and to do so with good accuracy and in a timely manner. There are people who know a lot more about NLP than I do; but even for the best folks in the field, semantic extraction is a difficult problem to work on.

Yes, we know how to do some useful stuff when it comes to semantic extraction. As a field, though, we’re really not all that good at this yet. It’s an area of very active academic research, particularly over the past two decades. Big players are putting big resources into it. We’ll keep at it, of course, because everybody knows that the business opportunities are huge.





The problem with NLP job descriptions

Following is a slightly edited version of a job opportunity which recently came across my desk. I’ve seen others like it.

A minimum of 4+ years of experience in software development with Java or Python

5+ years of implementing RESTful APIs, JSON, Web services

5+ years of Graph DB experience with TitanDB, Neo4J, or similar

3+ years of NLP experience

I’m going to be blunt: this is a ridiculous job description.

Why? Because it’s treating NLP as if it were just one more skill that a programmer can pick up, as if it were a database or a new programming language.

If you were trying to hire someone to submit insurance claims in a hospital billing office, you probably wouldn’t casually mention that the candidate should also have 3+ years experience as a heart surgeon.

NLP is a field, not a skill. To be decent at NLP, a candidate should ideally have a Master’s in the field, if not a Ph.D. Or, there are various ways that someone might manage to move into NLP if they have a strong background in other areas (particularly linear algebra).

As an employer, you’re going to pay a substantial premium for this skill set. If you’ve got a highly specialized employee who can do NLP, I don’t think it makes any business sense at all to also assign that person to also work in a commodity skill area such as web services. Nor should the specific choice of programming languages be your first concern. You’d be a fool to turn down an otherwise qualified NLP candidate because he or she has worked in Perl rather than Python.

Most tasks in NLP are simply too large in scale to be handled by an individual. It’s generally not the sort of effort that a single whiz kid can realistically take on. The field of NLP isn’t at the stage where there are convenient off-the-shelf solutions; there are multiple toolkits and libraries, but there are few people (if anyone) who can take that bucket of parts and singlehandedly hack together the sort of large-scale solutions that many businesses are looking for. Just gathering and labeling the data to train your models can be a major undertaking in itself.

I’ve seen a few good job opportunities which have stood out from the rest. These are ones where there’s a whole team of NLP specialists involved in some R&D effort. This kind of job looks more realistic. Of course, it means that an organization has to have the ability and desire to commit this level of resources to such an effort.

Be careful when you evaluate candidates. So much demand creates a pressure for people to declare themselves to be NLP specialists. It’s not like there’s any licensing authority. You might find someone who has read a book and has learned to do some things, but that sort of person is probably not going to have the depth of knowledge to solve the broad-scale problems that most businesses are interested in (although he or she might have some value as a junior member of a team). I haven’t actually seen a whole lot of this sort of problem, but it’s something to bear in mind.





Summary

From a recruiting standpoint, the market is a pretty bleak one. There is way too much demand for too few skilled people. Market forces will probably correct this over the next decade or so, but even the colleges and universities are still in the process of ramping up their NLP course offerings. Your best chance is to make sure that the employers are being realistic in what they are looking for.



