Chatbotdialogue management is designed to summarize the current dialogue state and determine what the system should do. It is typically divided into two sub-modules: “Dialogue State Tracking,” which is responsible for updating the dialogue state, taking as inputthe Natural Language Understanding modulederived user actions and past dialogue history, and outputting the dialogue state; and “Dialogue Policy Learning,” which determines system actions, taking the dialogue state as input and outputting system actions. The aforementioned “user actions, dialogue state, and system actions” can all be represented by an intent and a set of slot values.
Dialogue State Tracking
aims to update the dialogue state through “user actions” and “dialogue history.” The dialogue history may contain “multi-turn dialogue” content generated after the system asks follow-up questions due to insufficient information. Some requirements also reference User Profiles to supplement necessary “personalization information.” By reasoning and summarizing the above content, it converts into a simple dialogue state (one intent and a set of slot values), allowing the system to map the current dialogue state to a more complete Representation. To account for natural language ambiguity, speech recognition errors, or Natural Language Understanding module errors, methods can be further divided into 1-Best, which only considers the highest confidence user action, and N-Best, which considers multiple user actions and their confidence levels. CRF, RNN, and LSTM models are commonly used to track sequences.
Dialogue Policy Learning
aims to determine what the system should do based on the “dialogue state.” If the intent in the dialogue state falls within the services the system can provide, the system checks if the slot values are complete, then uses their content to query service APIs to obtain key answers or content. If the intent in the dialogue state is unclear, or if it matches a specific service but slot values are missing, the system should proactively ask the user, collecting sufficient information through multi-turn dialogue and dialogue state tracking. Finally, key answers or content are encapsulated into system actions, represented by one intent and a set of slot values, and provided to the Natural Language Generation module.
Implementation Methods for Dialogue Policy
- Rule-based methods establish system actions corresponding to user actions under various slot states by writing explicit rules. This method cannot handle uncertain states and requires manually writing a large number of rules, making it suitable only for simple scenarios in specific domains.
- Finite-State Machine (FSM) based methods can be divided into two schemes: “nodes represent slot states, edges represent system actions” and “nodes represent system actions, edges represent slot states.” Slot states can be categorized as present or absent, and system actions as asking for a slot or providing a final answer. To avoid low confidence, an action can be added to ask the user for secondary confirmation. Since the former causes the number of states to increase rapidly when slots are added, it is only suitable for data-driven approaches; if building manually, the latter is recommended. The advantage of using an FSM is that implementation is simple and easy to understand, but the disadvantage is that every state and action must be manually designed, which is not conducive to complex scenarios.
- Statistical-based methods typically use Markov Chains to represent the dialogue process as a decision process, where the system determines the next action in each dialogue state. The advantage of using Markov Chains is that one only needs to define slot states and system actions within the decision process to automatically learn state transition relationships. Reinforcement Learning and Online Learning can also be introduced during the process. The disadvantage is that it still requires manual design and is not conducive to complex scenarios.
- Deep Learning based methods take user actions and related features as input and output corresponding system actions to train deep neural network models. Deep Learning based methods require large amounts of training data to achieve effectiveness, which is currently difficult to satisfy in practical applications.
Specific Process of Dialogue Management
- the Natural Language Understanding moduleObtain the user dialogue “Recommend me a restaurant in Taipei.” At this point, the system detects the user intent and recognizes named entities, encapsulating the results into a user action (Intent=Recommend Restaurant, Location=Taipei), thereby mapping natural language to a simple semantic representation.
- Dialogue State Tracking ModuleUpdates the current dialogue state through the user action (Intent=Recommend Restaurant, Location=Taipei), then fills in common default values for the location, supplements personalized dining information via the User Profile, and finally outputs the dialogue state (Intent=Recommend Restaurant, Location=Taipei Gongguan, Taste=Likes Spicy Food).
- Dialogue Policy Learning ModuleAfter obtaining the dialogue state, it finds that the intent is within the services the system can provide, but the dining time is still missing, so the system should ask the user. Therefore, it outputs a system action (Intent=Ask about Empty Slot, Location=Taipei Gongguan, Taste=Likes Spicy Food, Time=Null).
- Natural Language Generation ModuleExecutes the system action, generating a question to ask the user about their desired dining time: “When would you like to dine?”
- the Natural Language Understanding moduleObtain the next turn of user dialogue “Tomorrow at noon,” detect intent and recognize named entities again, obtaining a user action (Intent=Unclear, Time=2020-03-30 12:00).
- Dialogue State Tracking ModuleReference the user action and dialogue history to update the current dialogue state (Intent=Recommend Restaurant, Location=Taipei Gongguan, Taste=Likes Spicy Food, Time=2020-03-30 12:00).
- Dialogue Policy Learning ModuleUse the dialogue state and dialogue history to collect necessary information for the restaurant recommendation service. After querying the service API to get the answer, encapsulate it into a system action (Intent=Recommend Restaurant, Location=Taipei Gongguan, Taste=Likes Spicy Food, Time=2020-03-30 12:00, Restaurant=Right Hand Restaurant, Type=Thai Cuisine).
- Natural Language Generation ModuleExecute the system action, generating a specific answer: “I suggest you go to Right Hand Restaurant in Taipei Gongguan tomorrow at noon to enjoy sour and spicy Thai cuisine.”
Future Development Directions
To address the widespread lack of training data for Deep Learning based dialogue management methods, the industry has attempted to use N-Shot Learning for training with small samples, or Zero-Shot Learning to transfer and complete existing models without any training data. Additionally, Reinforcement Learning and Online Learning are introduced into Markov Chain decision processes to establish reward/punishment and continuous learning mechanisms. Some scholars have also applied Generative Adversarial Networks (GAN) to Natural Language Processing, using SeqGAN combined with Reinforcement Learning to let two models compete against each other to learn the strongest dialogue strategy.
References
- POMDP-based Statistical Spoken Dialogue Systems: a Review
http://mi.eng.cam.ac.uk/~sjy/papers/ygtw13.pdf - Maximum Entropy Markov Models for Information Extraction and Segmentation
http://www.ai.mit.edu/courses/6.891-nlp/READINGS/maxent.pdf - Reinforcement Learning
https://en.wikipedia.org/wiki/Reinforcement_learning - One-Shot Learning
https://en.wikipedia.org/wiki/One-shot_learning - Zero-Shot Leaning
https://en.wikipedia.org/wiki/Zero-shot_learning - SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
https://arxiv.org/abs/1609.05473
