One highly sought after engineering role at major tech companies today is the natural language processing, or NLP, engineer.
NLP is a subset of machine learning which involves text and language recognition, rather than image analysis done with computer vision. Engineers hired for an NLP role could be responsible for implementations relating to tools like search recommendations, chatbots, and personalized text-based marketing.
While this role may be scalable to companies holding mass amounts of textual data, the hiring process is still competitive due to the shallow machine-learning talent pool.
In a 2016 survey of 50 international customer-care executives, McKinsey researchers asked about goals the executives hoped to achieve in the next five years. More than half of the respondents said that they wanted to use technology to create more self-service customer care options.
The study noted that these self-service options could include NLP-based virtual agents which can respond to text-based requests in a messaging thread or voice-based requests over the phone.
Despite the interest in NLP technology, McKinsey says that most of the executives surveyed are not able to hire someone at this time because they are lacking strategies needed to do so. The study further notes:
Two-thirds of our respondents said they were not prepared to invest in new technologies because they lack a clear strategy and think that such investments would not have an impact in less than 12 months.
For assistance in hiring an NLP leader or expert, a business may consider using a company like Toptal. Toptal claims to vet and help match experienced AI engineers, including those with NLP specialties, with companies around the world.
To get more insight on what it takes to find an NLP engineer, we spoke with Toptal’s Director of Engineering and AI Lead, Pedro Alves Nogueira.
Before Toptal, Alves Nogueira received a PhD in Artificial Intelligence and Human-Computer Interaction from the University of Porto in Portugal. He graduated summa cum laude. Prior to this, he served as a user experience and human-computer interaction researcher at several European and international projects, including on the Carnegie Mellon – Portugal Programme.
Looking at Company Needs
As a company enters the pre-hiring phase, Nogueira says they must try to answer the following questions:
- What type of data is accessible?
- Is there enough data for an NLP professional to enter the position? If not how will the company collect the data?
- What are the technical needs that an NLP engineer must be able to fulfill?
- What are the business-end goals that will be accomplished by the NLP engineer or a team of engineers?
Nogueira states the two types of stakeholders who are most likely to make the best hiring decisions could be:
- One tech-minded person who understands the data and has the strongest possible statistical or mathematical education: “Someone who understands the technical models and the aspects of putting them together. This includes production, getting access to the data, determining what type of access and what type of data. or determining processing power to crunch the numbers.”
- One business-side stakeholder, such as a leader in sales or marketing, who’s going to see a benefit from the candidate’s models
When these two people answer the above questions together, according to Nogueira, they will also be able to write a clear job description and make well-informed hiring decisions as a team.
At this phase in the hiring process, Nogueira says team integration and communication across the board is crucial. While business-end involvement is needed, having an ideal AI engineer from the team, or someone with a strong technical and mathematical background will be necessary. He elaborates:
If you’re not a very technical person, the whole interviewing process is going to be difficult. If you think traditional developers sometimes seem to speak a different language, then you’ve never spoken to an AI engineer in detail.
When hiring for an AI specialty, such as NLP, Nogueira says having an AI engineer on hand is ideal. However, someone with a strong math or science background should still be able to determine what is needed in an NLP-specific role.
Entering the Interview and Screening Process
As Nogueira noted for hiring a remote machine-learning engineer, the two people or teams that created the job description will want to conduct the interviews together.
This should be a key interview strategy because a strong NLP engineer, especially one of the first on a team, should be able to articulate their work and their accomplishments to both sides of a company. He goes on to say:
This person needs to be able to understand the business and to articulate the business needs from the stakeholder into his or her actual work. Then they need to communicate their findings to stakeholders in such a way that it is actionable, makes sense, and is interpretable. They need to think about how the business works as a whole so they know where to track value.
Nogueira says the earliest hires must be able to “bridge” the gap between their technology team and business teams when it comes to communication.
“You’re not going to get that type of information from shareholders or stakeholders.” he added. “You’re going to have to think about the data you have, what you can take from it, and what would be advantageous for the company.”
When it comes to discussing the specifics of NLP in an interview, Nogueira advises that the technically-savvy company representative should be able to draft NLP-related questions and determine if the responses are relevant.
Additional Screening
Along with business and technology-oriented interviews, some companies may wish to have an even stronger screening process when looking for high-priced AI engineers, such as those working with NLP.
Toptal only recommends and connects the top 3% of AI engineers that apply to be on their site with outside companies. To determine who is in the top 3% of its candidates, Toptal reports that its vetting process includes the following steps:
- Language and Personality Interviews: Face-to-face or video chats done to determine that the candidate is able to write and speak English “very well.” These initial interviews also help recruiters to determine what the candidate’s personality is like.
- In-Depth Skill Review: The candidate performs assessments that will test their problem-solving abilities. Further details on what these assessments look like or how long they take were not available.
- Live Screening: At this stage, an engineer applying with an AI specialty, such as NLP, will be interviewed by an engineer with a similar skillset. The interviewer will also challenge the candidate with live problem-solving exercises.
- Test Projects: The candidate is given a real-world scenario and real-world data and told to complete a task within a deadline of one to three weeks.
After the engineer is hired and begins getting matched with companies, Toptal says it keeps in contact with these companies to monitor the engineer’s work quality. While they note that they will not tolerate poor quality work, the site does not note what happens to the engineer when Toptal is notified about their work-related issues.
Below is a graphic of Toptal’s screening process, which the company models after a funnel:
Inquisitive Hires Could Bring on Future Engineers
In Nogueira’s role at Toptal, he says he has been tasked with interviewing engineers in machine learning specialties, such as NLP. He asked candidates to describe project’s they had worked on, as well as their NLP-related achievements. He elaborates:
I probably did around 100 interviews over a couple of months. One of my favorite parts involved just asking people about their past projects to see if they were passionate, and to see if they were truly knowledgeable about what they did.
He explains that in areas like NLP, companies should have a technically savvy person on hand at interviews to prevent hiring someone who has merely worked with AI tools rather than actually going through the time-consuming process of building their own models. According to Nogueira:
A person’s most successful project stories] are not necessarily the most interesting points. Sometimes an engineer will work for years on end and get really frustrated. Those were probably the problems or the times that they went through the craziest techniques and the weirdest approaches.
That’s a good test to know who’s most passionate about their work, as well as how resilient and how creative they are.
Nogueira says he placed hired NLP engineers who are able to best communicate their success stories on to his screening teams which are involved with conducting future interviews.
The best communicating, most inquisitive individuals with a solid track record passed the screening process. “Then, I invited them over to keep doing what I was originally doing with them,” he says. “So far, it’s worked out very well.
Closing Thoughts on Hiring an NLP Engineer
Regardless of whether a business hiring a first engineer or one of many, Nogueira advises that companies should look for the most communicative and inquisitive individuals, while also making sure they are technically savvy enough to manage their specialty, such as NLP.
One area where businesses may be most interested in looking into NLP could be for customer service. McKinsey’s customer-care survey, noted above, reported that at least 30% of customer service interactions are done through online messaging or chats.
McKinsey estimates this number will only grow as younger, more tech-savvy generations begin requiring customer service assistance more frequently. As a result, McKinsey suggests these types of trends may open the door to “artificial intelligence agents,” such as chatbots or automated customer service bots that can process voice requests. While these virtual agents could allow consumers to help themselves, the study suggests that they could also free up time for human agents that could be used for revenue generation.
This article was sponsored by Toptal, and was written, edited and published in alignment with our transparent Emerj sponsored content guidelines. Learn more about reaching our AI-focused executive audience on our Emerj advertising page.
Header Image Credit: Hobson Associates
