Ethan Lab Software Development · Product Management · Tech Reviews
Search
Back to research notes
Artificial Intelligence

[Conversational AI-2] Chatbot Reading Comprehension – Natural Language Understanding

Updated

A basic text-based chatbot framework includes three major modules: “Natural Language Understanding, Dialogue Management, and Natural Language Generation.” Some bots that interact with users via voice also require “Automatic Speech Recognition and Speech Synthesis” modules, such as the well-known Siri and Google Assistant. This issue of the AI column will provide a detailed introduction to the chatbot’s core “Natural Language Understanding” module.

What is Natural Language Understanding?

Natural Language Understanding (NLU) is the process of converting (mapping) natural language into a machine-readable semantic representation. It is the most difficult technology in Natural Language Processing (NLP). To enable machines to understand natural language, one must analyze “phonetics, phonology, morphology, syntax, semantics, and pragmatics,” which can be simply understood as follows:

  • Phonetics: How humans produce speech sounds
  • Phonology: How to spell out the pronunciation of natural language
  • Morphology: How to form words in natural language
  • Syntax: How to form sentences in natural language
  • Semantics: How to understand sentences in natural language
  • Pragmatics: How to use sentences in natural language

For machines, Natural Language Understanding may encounter the following problems:

  • Can errors generated by speech recognition be tolerated in Natural Language Understanding?
  • Chinese does not use spaces to separate words like English does; how can words be correctly segmented?
  • How to accommodate hundreds of natural language representations for the same meaning? For example: “I love you, I like you, I fancy you, you are my type,” etc.
  • The same sentence may have different meanings depending on the context (contextual situation); how can this be correctly determined?
  • What content do pronouns or omissions in a dialogue represent?
  • How to handle vocabulary that has not yet been learned? For example: “Is it hello, novel coronavirus.”

The Function of Natural Language Understanding in Chatbots

The functions performed by Natural Language Understanding in chatbots can be mainly divided into the following seven points:

  • User Intent Detection: Classifying the intent of user dialogues to determine what the user wants or plans to do. This can be divided into explicit intents (directly known classification, e.g., “Is the weather good today?”) and implicit intents (indirectly inferred classification, e.g., “Is it suitable to go out today?”). In both questions above, the user intent is “check the weather.”
  • Named Entity Recognition: Used to extract entities with specific meanings from user dialogues, such as proper nouns like “date, location,” to fill specific intent slots (Slot filling). For example, in “What is the weather in Taipei on February 25, 2020?”, the slot value for the date is “February 25, 2020,” and the slot value for the location is “Taipei”; the aforementioned user intent “check the weather” requires these two slot values for clarification.
  • Coreference Resolution: Determining the entities or events represented by pronouns in a dialogue, for exampleExamples of Winograd Schemas。
  • Ellipsis Recovery: Determining the content omitted by the user in a dialogue.
  • Sentiment Analysis: Identifying subjective information in user dialogues, such as positive or negative, or finding more complex states, such as happy, angry, sad, etc.; this can make machine-human interactions warmer.
  • Intent Confirmation: When the confidence level of intent recognition is insufficient, ask the user to confirm again.
  • Rejection Identification: When the recognized intent is outside the service scope, involves sensitive content, or has too low confidence, the system can refuse to reply.

Different types of chatbots (please refer toTypes and Comparison of Chatbots), have different emphases in Natural Language Understanding technology, as follows:

1. Question Answering System:

Focuses on identifying information words in questions, such as question words (Who, What, Where, When, Why, How), focus words, topic words, central verbs, etc. This differs slightly from Natural Language Understanding and leans more towards the scope of Natural Language Processing. The four mainstream methods are template matching, statistics-based, deep learning-based semantic analysis, and deep learning-based end-to-end generation (this method is currently in the research stage). The main evaluation metrics are Recall, Precision, and F-Score.

2. Task-Oriented Dialogue System

Focuses on converting natural language into machine-readable semantic representations, such as dividing into User Intent and Slot Value. Specific tasks include Chinese word segmentation, part-of-speech tagging, named entity recognition, coreference resolution, and syntactic parsing. The three mainstream methods are rule-based, statistics-based, and deep learning-based intent recognition and slot filling. The main evaluation metrics are intent classification Accuracy and slot filling F-Score.

3. Chit-Chat Dialogue System

Focuses on personalization and sentiment analysis (using Natural Language Processing to identify subjective information in dialogues, such as positive or negative). The two mainstream methods are dialogue library retrieval and deep learning-based end-to-end generation. The main evaluation metrics are word overlap rate and vector distance.

Implementation Methods for Natural Language Understanding

As mentioned above, Natural Language Understanding can be viewed as a classification and sequence labeling problem, typically divided into three methods: rule-based, statistics-based, and deep learning-based:

  • Adopting a rule-based method has the advantage of being easy to adjust and correct, and does not rely on training data. The disadvantage is that as scenarios increase, the number of rules increases significantly, making it difficult to maintain.
  • Adopting a statistics-based method has the advantage of high system robustness and ease of maintenance. The disadvantage is that the trained models are difficult to explain and correct. Support Vector Machine (SVM) or Adaptive Boosting is commonly used for intent classification, while Hidden Markov Model (HMM) or Conditional Random Field (CRF) is used for word segmentation and entity recognition.
  • Adopting a deep learning-based method has the advantage of best performance, high system robustness, and ease of maintenance. The disadvantages are the need for large amounts of training data, high spatial and temporal complexity of the models, and near-impossibility of explanation and correction, acting like a black box. Attention-based RNN and LSTM models are commonly used for tasks such as intent detection and slot filling.

Other Uses of Natural Language Understanding Related Technologies

The Natural Language Processing technologies involved in Natural Language Understanding can be used for purposes other than chatbots:

  • Spam Detection: Analyzing large amounts of data to induce features, helping users filter spam emails, such as the spam folder in Gmail.
  • Search Engine Suggestions: Proactively recommending search content by predicting the words users input, such as the recommended search list that appears when typing a few words in Google Search.
  • Machine Translation: Translating one natural language into another, such as the English-to-Chinese function in Google Translate.
  • Automatic Summarization: Analyzing one or more articles to automatically generate a summary, commonly applied in news briefs.

References

Research notes