NLP for eCommerce Search – Current Challenges and Future Potential

Raghav Bharadwaj

Raghav is serves as Analyst at Emerj, covering AI trends across major industry updates, and conducting qualitative and quantitative research. He previously worked for Frost & Sullivan and Infiniti Research.

Robbie Allen from Automated Insights - The Use-Cases of Natural Language Generation 2

Episode summary: In this week’s interview on the AI in Industry podcast, we speak with Amir Konigsberg, the CEO of Twiggle, about the future of product search – and how eCommerce and retail brands can use natural language processing (NLP) to improve their user experience.

Amir explains some of the factors that make eCommerce product search challenging, and the artificial intelligence approaches that can improve it today and within the next five years.

Subscribe to our AI in Industry Podcast with your favorite podcast service:

Guest: Amir Konigsberg, CEO of Twiggle

Expertise: Business operations strategy, user experience technologies , human machine interface (HMI)

Brief recognition: Amir was a research fellow of the Max Planck Institute in Berlin, Germany and went on to work with Google in business operations strategy for Israel from 2005-2008. He has also previously served as a lecturer for the Hebrew University in Israel, and as a visiting research fellow at Princeton University. He has also worked with General motors leading projects in user experience technologies and human machine interfaces.

Current affiliations: Amir is currently on the management team of Israel Brain Technologies. In addition to serving as CEO at Twiggle, he is also an advisory board member for Seematics Systems.

(Readers interested in eCommerce and retail chatbot use cases can learn more about present and future use-cases for artificial intelligence for retail in our full article on that topic.)

Big Idea

Today’s eCommerce search engines can index a large number of product names or descriptions to aid product search, but they don’t always deliver relevant results.

According to Amir, this traditional search process needs continuous improvement through the addition of contextual awareness to give accurate results. Natural language processing (NLP) can be used to add this ‘awareness’ to better understand both the search queries and the products in a typical eCommerce application. A few examples are discussed below:

  • If a user phrased his search query as “blue dress-shirt”, the search engine might deliver results about dresses and shirts but not necessarily a dress-shirt, because the concept of a dress shirt is something the engine might not know unless NLP based contextual awareness is integrated.
  • For a search like “bike helmet” most current eCommerce search engines will deliver results for bikes too. NLP can help the search engine understand that the word ‘bike’ is describing the word ‘helmet’
  • A search query for ‘sleeveless shirt’ would involve understanding context of the words, and telling the engine what you don’t want, this type of negation in search is very hard for traditional search engines without NLP.

Amir mentions that successful integration of NLP into online product search is still challenging.

“For example in a typical retail eCommerce application, this would involve getting an algorithm to take all the data about every product being sold, structure and normalize it, and then find all linguistic attributes in which you can describe a particular product. What the challenge here is really leveraging NLP technologies to put the burden on the search engine and not on the consumer to make the experience natural”

On the front-end, successful integration of NLP for eCommerce search  would also involve getting the NLP platform to understand specific language used in a retail scenario. According to Amir:

“This is only possible in a specific domain like retail; it cannot be achieved in a generic sense across industries with today’s technology”

In the near future (five year forecast period) Amir thinks that online product search in the eCommerce and retail domain will evolve in the following ways:

  • Expect that Siri-like ‘assistance’ devices will be everywhere within 5-6 years. But search experiences in retail will be ‘assistive’ (asking a question and getting accurate answers) and more about guiding users to where they want to go.
    • For example: A customer could say “I want a retro look for winter” and the assistant would understand the context for ‘retro’ and deliver search results based on purchase history for matching shirts and pants in the user’s size.
  • Online product search will evolve in a way where the context understanding is integrated with the search engine letting humans converse with them in a way that is very ‘natural’ in a ‘verticalized’ environment.
    • For example: Customers in fashion retail have a different way of phrasing requests as opposed a customer in the home appliances space. NLP platforms of the future will be a lot better at contextually understanding of these nuances. Other sector-specific NLP use-cases (in automotive, in personal assistants, etc) have similar requirements and will also need to evolve.
  • In the future product search will take into account personal information, context on the user and context on the history to deliver highly accurate search results. Currently, eCommerce leaders like Amazon have easy access to this kind of user history data, and other smaller companies may not.
    • Small businesses will be able to compete with the search quality of Amazon in the near future with the help of NLP-powered search engines that do not need repetitive shopping behavior to come up with similar results.

According to Amir, in the future customers could phrase search queries like “I’m looking for a laptop for business use, it must be lightweight and I don’t want to spend more than $1000 dollars” and get 1-2 highly accurate search results. This level of accuracy will be achieved by taking into account who you are, what you are most likely to buy based on purchase history.

Readers interested in the relative performance of Siri, Alexa, and other conversational interfaces should see our full article on chatbots comparisons between Google, Apple, and others.

Interview Highlights with Amir from Twiggle

  • (1.46) What are some of the initial hurdles in making eCommerce search work well?
  • (9.57) What are some examples of areas where NLP techniques might help to outperform traditional eCommerce search operations?
  • (18.32) What is product search going to look like half a decade from now, specifically, how will it evolve?

Subscribe to our AI in Industry Podcast with your favorite podcast service:


Header image credit: Adobe Stock