The ability for computers to understand online shoppers’ intent has been an elusive goal. There are several reasons for this, but the main culprit is the stateless nature of today’s search engines. This means a shopper must accurately specify all of the key attributes for a desired item inside a search box, in a single instance. Anyone who shops online knows how difficult and frustrating it can be to refine a search and explore for items at the same time.
What is needed is a far more natural and conversational online shopping approach and experience. How chatbots bridge the gap between the stateless search engine query and a shopper’s actual intent is a challenge that has many of eBay engineers’ undivided attention. And for good reason. Crossing this chasm will amount to a scientific accomplishment equivalent to the Holy Grail of e-commerce.
The Future Has Arrived
Now that voice interfaces have finally started to shine and are becoming ubiquitous, shoppers will expect the new technology to mesh seamlessly with the software bots designed to search the Internet for deals on products and services. For these types of personalized, digital shopping assistants to be more than a passing fad, they must possess the brainpower to engage in intelligent, conversational exchanges — using either speech or text-based search. And that’s where Natural Language Understanding (NLU) comes into play.
eBay engineers are busy baking NLU capabilities into eBay ShopBot — the smart personal shopper that helps consumers find the items they want to buy from a marketplace of approximately 1.1 billion live listings. eBay ShopBot is available today as a public beta on Facebook Messenger and overtime will be added to other popular platforms. The eBay team is working to enhance conversational search and make the experience like talking with a friend.
Engineers are teaching machines to understand the true nature of buyers’ intent. If a shopper is looking for a “red cocktail dress,” the bot needs to act like a helpful human sales clerk — shepherding the buyer to his/her version of perfect. To make that leap, the bot needs to recognize vague clues and nuances including acronyms. It also must have the smarts to translate made-up or misspelled words. Why? Because this is the reality of how humans use search today. It is a type of millennial shorthand that is becoming the new lingua franca for e-commerce.
NLU Secret Sauce
There has been a lot of buzz in the chatbot community about how the eBay ShopBot team extracts important information from naturally-expressed phrases to provide better search results. eBay engineers have developed key capabilities that help decrypt the meaning of spoken or written words, phrases and sentences. eBay ShopBot understands full, conversational language and it uses past interactions to provide context and find richer meaning in conversational-style searches.
This is possible by leveraging algorithms that learn automatically from eBay’s extensive inventory and user queries.These algorithms are the backbone of the NLU technology stack that include: spelling correction, intent detection, named-entity recognition, information extraction and knowledge inference. The technology stack converts strings of sentences into a structured query to determine what the eBay ShopBot says and shows the user. Technologies such as word disambiguation, coreference resolution and sentiment analysis will further improve contextual processing in NLU and create a chatbot that gets more savvy and sophisticated over time.
NLU in Action
Here’s an example of how NLU drives eBay ShopBot’s effectiveness. In the following query: “Can you show me brown leather Coach messenger bags under $100?” Shopping is identified as the user’s primary intent. eBay ShopBot then uses a hybrid approach leveraging deep learning and syntactic dependency parsing to extract relevant information related to the shopper’s intent. In the example above, NLU identifies the object of interest as a messenger bag and the target price range to be between $0 and $100. A named-entity recognition component, trained on eBay queries, is used to identify brown as the color, leather as the material, and Coach as the brand.
Once the intent, object and characteristics of the object are known, the data is mapped to eBay inventory using a Knowledge Graph (KG). The KG encapsulates shopping behavior patterns on eBay to bridge the gap between the structured query and behavior data. In other words, the KG helps figure out the best follow-up questions to ask in order to find the best results in the least amount of time.
In chat applications, no one wants to type long messages. So queries tend to be less descriptive, such as: “I am looking for bags.” In this example, shopping is identified as the intent. The object of interest is bags. However, some important characteristics of the bag are unknown. To help identify the missing ingredients, KG identifies the most likely shopping categories for bags such as Handbags, Laptop Bags, and Backpacks. It presents these options to the user and asks that the best category be selected. The query subject is also remembered for all follow-up questions (brand, color, style and material).
ShopBot uses its Knowledge Graph to understand user requests and generate follow-up questions to refine requests before searching for the items in eBay’s inventory. In a search query for “bags” for example, purple nodes represent “categories,” green “attributes” and pink are “values” for those attributes.
Context is Key
Humans have the ability to understand when a conversation switches context or topic, but to achieve that with a machine is not that easy. NLU in eBay ShopBot analyzes every query in the context of the current dialogue. For example, if the user types “Black” in the chat window while searching for messenger bags, NLU remembers that the user was searching for messenger bags and understands that black is a color that overrides any previously selected color. As a result, eBay ShopBot will present black messenger bags to the user. The user can also change their search at any time by starting a new query, even without using explicit commands like new search. eBay ShopBot NLU leverages context to understand that the user mission, intent or both, have changed.
eBay is committed to driving a more humanized search experience. The company has one of the world’s largest commerce data sets that provides unique insights into online commerce. If you are interested in helping us shape the future of e-commerce, we would love to hear from you. Please check our open positions .
And remember to say hello to eBay ShopBot and start shopping with your personal assistant today.
The following eBay scientists and engineers are the authors and great minds behind this post: Amit Srivastava, Sanjika Hewavitharana, Ajinkya Kale, and Saab Mansour.
Editor's Note: This post appeared originally on Medium.