Dialog Flow Generation To Constrain LLM-Based Chatbots
This paper presents an unsupervised method for automatically generating domain-specific dialog flows to guide and constrain LLM-based chatbots.
Introduction
The chart below highlights a key tension in chatbot development: the balance between flexibility and control.
When more control is embedded into the conversation flow, it often leads to tighter constraints, making the interaction less flexible.
Conversely, as the conversation becomes more flexible from the user’s perspective, there is less control over guiding the dialogue toward achieving specific business objectives.
This trade-off is a critical consideration in designing effective chatbots.
Hence, astute chatbot development frameworks strive to find an optimal balance between flexibility and control.
By doing so, they aim to provide users with a more natural and adaptable conversational experience while still guiding interactions toward desired business outcomes.
These frameworks often incorporate adaptive learning mechanisms that allow chatbots to adjust their responses dynamically without sacrificing the overall direction of the conversation.
The goal is to ensure that chatbots remain user-friendly while effectively driving business goals.
Ultimately, this balance enhances both user satisfaction and the chatbot’s effectiveness in fulfilling its intended purpose.
Chatbots are Domain Specific
Yes, there are general-purpose conversational UIs like ChatGPT, HuggingChat, Cohere Coral, Alexa, and Google Home.
These systems constantly strive to establish context by understanding the user and their specific domain. By grasping this context, they can better comprehend and address the conversation, ultimately enhancing the quality and relevance of their responses.
An unsupervised method for automatically generating domain-specific dialog flows that can constrain LLM-based chatbots. Source
However, most chatbots are domain specific, servicing the products and services of a specific organisation or domain.
OK, so considering domain together with flexibility/control, there is obviously an immense opportunity to have LLM-driven dialogs and really the power that LLMs bring…
Recent research addresses the challenge of keeping LLM-driven dialog systems focused within specialised domains, a crucial need in applications like healthcare and customer service.
Due to their broad generalisation capabilities, these chatbots often risk providing inaccurate or irrelevant responses. To tackle this, the authors propose an unsupervised method for automatically generating domain-specific dialog flows that can constrain LLM-based chatbots.
They present two variations of these dialog flows, depending on the availability of in-domain conversation data.
Through both human and automated evaluations across various dialog domains, the study shows that these data-guided dialog flows offer better domain coverage, reducing the need for extensive manual crafting.
The design of dialog schemas thus has significant manual overhead for developers, resulting in scalability and coverage limitations. Source
The figure illustrates the process of intrinsic flow induction. It begins with the generation of an initial flow, which is then refined through a series of feedback, update, and clean-up stages. This is achieved via detailed prompts.
Clusters
Considering the image below, something which is note worth, is the fact that utterances are grouped in clusters which are semantically similar.
Intents
By merging semantically similar clusters that show significant overlap in action intent — identified through cosine similarity between their labels — this approach reduces redundancy in action labels by consolidating clusters with similar actions.
Hence this process of detecting strong signals, surfaces the relevant intents for this conversation.
Flow Generation
The process for generating dialog flows closely mirrors the intrinsic dialog flow generation.
Initially, representative action labels and sample conversations from the domain are included in the prompt for the initial flow generation. Subsequently, feedback, updates, and clean-up steps are applied to refine the dialog flow.
The intrinsic flow method produces broad, general dialog flows but may lack domain-specific patterns observed in real-world conversations.
Conversely, relying solely on domain-specific dialog instances might limit the flow’s scope due to their variability. To address this, a hybrid approach is used, combining the broad coverage of intrinsic flows with the detailed focus from domain-specific data.
This merging process involves prompting GPT-4 to retain distinctive features from the intrinsic flow while eliminating redundant elements, resulting in what is termed the data-guided flow.
Intrinsic Flow Example
Below is an example of an intrinsic flow called city_info_code.txt . Notice how the user intent is defined and encoded. Then each of these intents are mapped to a bot flow which is also coded.
For example U1 goes to B1, and so on.
B0[Bot: Greeting and Asks for City Information Inquiry] --> U1[User: Wants to Know About Attractions]
B0 --> U2[User: Inquires About City Services]
B0 --> U3[User: Asks About Local Regulations]
B0 --> U4[User: Interested in City Events]
B0 --> U45[User: Asks for Personalized Recommendations]
B0 --> U46[User: Inquires About Local Culture and History]
B0 --> U47[User: Wants Info on Dining and Shopping]
B0 --> U48[User: Seeks Accessibility and Accommodation Info]
U1 --> B1[Bot: Provides Information on Attractions]
U2 --> B2[Bot: Details City Services]
U3 --> B3[Bot: Explains Local Regulations]
U4 --> B4[Bot: Shares Information on Upcoming Events]
U45 --> B5[Bot: Gives Personalized Recommendations]
U46 --> B6[Bot: Discusses Local Culture and History]
U47 --> B7[Bot: Informs About Dining and Shopping]
U48 --> B8[Bot: Advises on Accessibility and Accommodation]
B1 --> U9[User: Requests Further Details or Ready to Explore]
B2 --> U9
B3 --> U9
B4 --> U9
B5 --> U9
B6 --> U9
B7 --> U9
B8 --> U9
U9 --> B9[Bot: Provides Additional Info or Encourages Exploration]Data Flow Example
Below is an example of a data flow, called alarm set code, which should be a familiar conversation to many.
Again, notice the intents being defined upfront, and how the bot dialogs are assigned to different user intents. There is also the notion of sub or nested intents.
B0[Bot: Greeting] --> U1[User: Express need for alarm assistance]
U1 --> B1[Bot: Prompt for alarm action]
B1 --> U2[User: Set alarm]
B1 --> U3[User: Edit alarm]
B1 --> U4[User: Delete alarm]
B1 --> U5[User: Copy alarm]
U2 --> B2[Bot: Set Alarm]
U3 --> B3[Bot: Ask for alarm details]
U4 --> B4[Bot: Ask which alarm to delete]
U5 --> B9[Bot: Ask details for copying alarm]
B2 --> U6[User: Confirm alarm set]
B2 --> U7[User: Request to set repeat]
U6 --> B7[Bot: Confirm Completion]
U7 --> B8[Bot: Set repeat and confirm]
B3 --> U8[User: Specify alarm edit details]
B4 --> U9[User: Specify alarm to delete]
B9 --> U14[User: Confirm copy details]
U8 --> B5[Bot: Perform edit and confirm]
U9 --> B6[Bot: Delete alarm and confirm]
U14 --> B10[Bot: Copy alarm and confirm]
B5 --> U10[User: Confirm edit]
B6 --> U11[User: Request additional deletion]
B6 --> U12[User: End conversation]
B10 --> U15[User: Request further changes]
B10 --> U16[User: End conversation]
U10 --> B7
U11 --> B6
U12 --> B11[Bot: Expressing gratitude and farewell]
U15 --> B3
U16 --> B11
B8 --> U13[User: End Conversation]
U13 --> B11Example Conversations
Below is a short snippet of the example conversations used to generate the alarm code reset flow. The complete training file has about 700 conversations. Each line is a conversation defined by the utterances, the session id, domain and task.
{"session_id": "298663b0", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "i need to edit an alarm", "ok which one?", "the \"walk dog\"", "what would you like to do with it?", "change to \"feed cat\"", "changing. do you want to apply this to other alarms with this name?", "yes", "ok", "thanks", "yes"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "9fefc2ac", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I would like to edit a feature on my alarm.", "Alright, what would you like to edit?", "I would like to cancel all my alarms for the next week, but could you first list them all before I do anything?", "Sure. You have 5 alarms for next week. Monday at 8 AM, Tuesday at 9 AM, Wednesday at 10 AM, Thursday at 11 AM, and Friday at 12 PM. Would you like to cancel all of the alarms for next week?", "Alright, thanks. Could you cancel all my alarms?", "Sure. This action cannot be undone once completed. Would you like to proceed with deleting all of your alarms for next week?", "I would like to confirm my cancellation of all my alarms for next week.", "Your alarms for next week have been deleted.", "Very good. Thank you very much!", "You're welcome!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}My Observations
The process of flow creation starts with generating an initial flow by using a large language model (LLM). The level of automation can work and serves well for solving for the blank canvas problem or blank canvas paralysis.
However, starting out with existing conversations is a much more astute approach. Making use of existing dialog turns between customers and service representatives leverages impactful and meaningful conversations.
I guess someone can follow the same methodology and only replace the initial LLM dialog generation with real-world conversations.
The aim of this approach is premised on only involving humans in a weak supervision role, hence elevating the burden on designers and builders.
The dialogs are also premised on a graph approach with nodes and conditional edges.
This tool is ideal for making use of existing conversations, creating semantically similar clusters, and creating or generating flows which can be used as a starting point when developing a chatbot.
The flows can be used as a starting point for designing and building chatbots and creating prototype flows to build from.
I’m not sure what the best why is to implement the generated flows…having a tool to convert the ASCII into a visual representation or some kind of GUI will be helpful.
Or some kind of development framework which can interpret and ingest the text based design.
I’m currently the Chief Evangelist @ Kore AI. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.