Conditional logic helps analyze the generated data’s relevance based on specific criteria. This involves understanding the relevance, details, and how the data is fetched and processed. Can reduce the hallucination probability.
Table of Contents
Pre-requisites
- Git hub code here
- Go through my previous posts to understand this blog better:
- I use Google Colab to make use of the Google Colab resources, You can use a Python script to run the code
Use case: Symptoms and diagnosis Medical chatbot Document retrieval is presented to the doctor for further analysis
What’s New in this article?
- Load and use the hugging face dataset
- Use hugging face embeddings
- Extensive use of conditionals to check the relevance of the documents and detect hallucinations.
- Creating a fallback option using initialize_agent to run the Pubmed query
Flow design and diagram
The flow of the code is as follows:
- Retrieve the dataset from the vector DB
- Grade the documents: Check whether the document is relevant or not This information is passed on to the next step
- Generate: generates the answer based on the question. The generated answer is classified as useful, supported, and not useful. Not supported will trigger generate again. The useful one is evaluated further in the next step
- Grounded? This step checks whether the generated answer is grounded to the data in the RAG. This step evaluates whether the model is hallucinating. This can be a game changer for certain scenarios where the model hallucinates.
- If the answer is in the RAG, then once again the model checks whether the generated answer answers the question. If yes then the answer is accepted. If not it triggers a fallback approach that calls the PubMed query to get the answers.
- If the answer is not in the RAG, then the fallback method will be triggered which uses PubMed to answer the query and send it back to the user.
LangGraph
The graph is designed to retrieve, and grade documents, generate answers, and classify them into useful and not useful. In scenarios where it is not useful, the fallback mechanism is activated and results are used as a final answer.
Code guide
- grade_documents: Evaluate whether the document contains the keyword(s) or semantic meaning related to the user question, and grade it as relevant, if not grade it as not relevant. This adds the documents that are relevant into a list called filtered_docs. The empty list means not relevant. It is a binary answer yes or no.
- GradeHallucinations: Evaluate whether the documents are grounded with the set of retrieved facts or not.
- GradeAnswer: Does the generated answer resolve the question? yes or not answer.
- The graph state is as below:
- fallback: This is an agent that calls PubMed to retrieve answers in case the symptoms the patients have are not in the existing dataset.
Results:
Query 1: “I have heartburn and indigestion. I vomit everything I eat. Food gets stuck in my throat, making it difficult for me to swallow. I have a pain in my upper abdomen”
---RETRIEVE---
START
I have heartburn and indigestion. I vomit everything I eat. Food gets stuck in my throat, making it difficult for me to swallow. I have a pain in my upper abdomen
documents
[Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I often vomit whatever I eat and have difficulty swallowing food. I have a nagging pain in my upper abdomen. I have a hard time sleeping because of the pain."'), Document(metadata={'output_text': 'peptic ulcer disease'}, page_content='"I have a sour acidic taste in my mouth. I have frequent belching and burping. I have a feeling of pressure or fullness in my upper abdomen that last for a few hours. I have a pain in my upper abdomen that is worse when I eat."'), Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It\'s worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up."'), Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I regularly throw up everything I eat, and swallowing is difficult for me since food gets stuck in my throat. I\'m experiencing pain in my upper abdomen."')]
"Node 'retrieve':"
{'documents': [Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I often vomit whatever I eat and have difficulty swallowing food. I have a nagging pain in my upper abdomen. I have a hard time sleeping because of the pain."'),
Document(metadata={'output_text': 'peptic ulcer disease'}, page_content='"I have a sour acidic taste in my mouth. I have frequent belching and burping. I have a feeling of pressure or fullness in my upper abdomen that last for a few hours. I have a pain in my upper abdomen that is worse when I eat."'),
Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It\'s worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up."'),
Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I regularly throw up everything I eat, and swallowing is difficult for me since food gets stuck in my throat. I\'m experiencing pain in my upper abdomen."')],
'question': 'I have heartburn and indigestion. I vomit everything I eat. Food '
'gets stuck in my throat, making it difficult for me to swallow. '
'I have a pain in my upper abdomen'}
'\n---\n'
---CHECK DOCUMENT RELEVANCE TO QUESTION---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT NOT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---ASSESS GRADED DOCUMENTS---
---DECISION: GENERATE---
"Node 'grade_documents':"
{'documents': [Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I often vomit whatever I eat and have difficulty swallowing food. I have a nagging pain in my upper abdomen. I have a hard time sleeping because of the pain."'),
Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It\'s worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up."'),
Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I regularly throw up everything I eat, and swallowing is difficult for me since food gets stuck in my throat. I\'m experiencing pain in my upper abdomen."')],
'question': 'I have heartburn and indigestion. I vomit everything I eat. Food '
'gets stuck in my throat, making it difficult for me to swallow. '
'I have a pain in my upper abdomen'}
'\n---\n'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---
---GRADE GENERATION vs QUESTION---
---DECISION: GENERATION ADDRESSES QUESTION---
"Node 'generate':"
{'documents': [Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I often vomit whatever I eat and have difficulty swallowing food. I have a nagging pain in my upper abdomen. I have a hard time sleeping because of the pain."'),
Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It\'s worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up."'),
Document(metadata={'output_text': 'gastroesophageal reflux disease'}, page_content='"I have heartburn and indigestion. I regularly throw up everything I eat, and swallowing is difficult for me since food gets stuck in my throat. I\'m experiencing pain in my upper abdomen."')],
'generation': 'You may be experiencing symptoms of gastroesophageal reflux '
'disease (GERD). It is characterized by heartburn, indigestion, '
'vomiting after eating, difficulty swallowing, and upper '
'abdominal pain. It is advisable to consult a healthcare '
'professional for proper diagnosis and treatment.',
'question': 'I have heartburn and indigestion. I vomit everything I eat. Food '
'gets stuck in my throat, making it difficult for me to swallow. '
'I have a pain in my upper abdomen'}
'\n---\n'
('You may be experiencing symptoms of gastroesophageal reflux disease (GERD). '
'It is characterized by heartburn, indigestion, vomiting after eating, '
'difficulty swallowing, and upper abdominal pain. It is advisable to consult '
'a healthcare professional for proper diagnosis and treatment.')Query 2: “I have been having migraines and headaches. I can’t sleep. My whole body is shaking and shivering. I feel dizzy sometimes.”
---RETRIEVE---
START
I have been having migraines and headaches. I can't sleep. My whole body is shaking and shivering. I feel dizzy sometimes.
documents
[Document(metadata={'output_text': 'drug reaction'}, page_content='"I have been having migraines and headaches. I can\'t sleep. My whole body is shaking and shivering. I feel dizzy sometimes."'), Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. My whole body shakes and twitches. Sometimes I feel lightheaded."'), Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. I\'ve been shaking and shivering all over. Sometimes I feel lightheaded."'), Document(metadata={'output_text': 'migraine'}, page_content='"I have been having frequent headaches that are so bad that they make me throw up. I also have blurred vision, a sore neck, and anxiety."')]
"Node 'retrieve':"
{'documents': [Document(metadata={'output_text': 'drug reaction'}, page_content='"I have been having migraines and headaches. I can\'t sleep. My whole body is shaking and shivering. I feel dizzy sometimes."'),
Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. My whole body shakes and twitches. Sometimes I feel lightheaded."'),
Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. I\'ve been shaking and shivering all over. Sometimes I feel lightheaded."'),
Document(metadata={'output_text': 'migraine'}, page_content='"I have been having frequent headaches that are so bad that they make me throw up. I also have blurred vision, a sore neck, and anxiety."')],
'question': "I have been having migraines and headaches. I can't sleep. My "
'whole body is shaking and shivering. I feel dizzy sometimes.'}
'\n---\n'
---CHECK DOCUMENT RELEVANCE TO QUESTION---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---ASSESS GRADED DOCUMENTS---
---DECISION: GENERATE---
"Node 'grade_documents':"
{'documents': [Document(metadata={'output_text': 'drug reaction'}, page_content='"I have been having migraines and headaches. I can\'t sleep. My whole body is shaking and shivering. I feel dizzy sometimes."'),
Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. My whole body shakes and twitches. Sometimes I feel lightheaded."'),
Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. I\'ve been shaking and shivering all over. Sometimes I feel lightheaded."'),
Document(metadata={'output_text': 'migraine'}, page_content='"I have been having frequent headaches that are so bad that they make me throw up. I also have blurred vision, a sore neck, and anxiety."')],
'question': "I have been having migraines and headaches. I can't sleep. My "
'whole body is shaking and shivering. I feel dizzy sometimes.'}
'\n---\n'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---
---GRADE GENERATION vs QUESTION---
---DECISION: GENERATION ADDRESSES QUESTION---
"Node 'generate':"
{'documents': [Document(metadata={'output_text': 'drug reaction'}, page_content='"I have been having migraines and headaches. I can\'t sleep. My whole body is shaking and shivering. I feel dizzy sometimes."'),
Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. My whole body shakes and twitches. Sometimes I feel lightheaded."'),
Document(metadata={'output_text': 'drug reaction'}, page_content='"I\'ve been having headaches and migraines, and I can\'t sleep. I\'ve been shaking and shivering all over. Sometimes I feel lightheaded."'),
Document(metadata={'output_text': 'migraine'}, page_content='"I have been having frequent headaches that are so bad that they make me throw up. I also have blurred vision, a sore neck, and anxiety."')],
'generation': 'The symptoms you are experiencing, such as migraines, '
'headaches, shaking, shivering, and dizziness, could be '
'indicative of a drug reaction. It is important to consult a '
'healthcare professional for proper evaluation and treatment.',
'question': "I have been having migraines and headaches. I can't sleep. My "
'whole body is shaking and shivering. I feel dizzy sometimes.'}
'\n---\n'
('The symptoms you are experiencing, such as migraines, headaches, shaking, '
'shivering, and dizziness, could be indicative of a drug reaction. It is '
'important to consult a healthcare professional for proper evaluation and '
'treatment.')Query 3: (Fallback) “I have a fever, chills, and skin rashes”
---RETRIEVE---
START
I have fever, chills and skin rashes
documents
[Document(metadata={'output_text': 'malaria'}, page_content='"I have a high fever, severe itching, chills, and vomiting. I also have a headache and have been perspiring a lot. Additionally, I have been feeling nauseous and my muscles are sore."'), Document(metadata={'output_text': 'malaria'}, page_content='"I have a fever, chills, and severe itching. I also have a headache and am sweating a lot. I feel awful and have nausea and muscle aches."'), Document(metadata={'output_text': 'malaria'}, page_content='"I\'ve been experiencing a high fever, vomiting, chills, and severe itching. I also have a headache and am sweating profusely. My discomfort has also been accompanied by nausea and muscle pain."'), Document(metadata={'output_text': 'malaria'}, page_content='"I have a high fever, severe itching, chills, and vomiting. I also have a headache and have been perspiring a lot. Additionally, I have been nauseous and my muscles hurt."')]
"Node 'retrieve':"
{'documents': [Document(metadata={'output_text': 'malaria'}, page_content='"I have a high fever, severe itching, chills, and vomiting. I also have a headache and have been perspiring a lot. Additionally, I have been feeling nauseous and my muscles are sore."'),
Document(metadata={'output_text': 'malaria'}, page_content='"I have a fever, chills, and severe itching. I also have a headache and am sweating a lot. I feel awful and have nausea and muscle aches."'),
Document(metadata={'output_text': 'malaria'}, page_content='"I\'ve been experiencing a high fever, vomiting, chills, and severe itching. I also have a headache and am sweating profusely. My discomfort has also been accompanied by nausea and muscle pain."'),
Document(metadata={'output_text': 'malaria'}, page_content='"I have a high fever, severe itching, chills, and vomiting. I also have a headache and have been perspiring a lot. Additionally, I have been nauseous and my muscles hurt."')],
'question': 'I have fever, chills and skin rashes'}
'\n---\n'
---CHECK DOCUMENT RELEVANCE TO QUESTION---
---GRADE: DOCUMENT NOT RELEVANT---
---GRADE: DOCUMENT NOT RELEVANT---
---GRADE: DOCUMENT NOT RELEVANT---
---GRADE: DOCUMENT NOT RELEVANT---
---ASSESS GRADED DOCUMENTS---
---DECISION: ALL DOCUMENTS ARE NOT RELEVANT TO QUESTION, TRANSFORM QUERY---
"Node 'grade_documents':"
{'documents': [], 'question': 'I have fever, chills and skin rashes'}
'\n---\n'
---Fallback to PUBMED QUERY---
> Entering new AgentExecutor chain...
Question: I have fever, chills and skin rashes. What could be the cause?
Thought: I should search for potential causes that include the symptoms of fever, chills, and skin rashes.
Action:
```
{
"action": "PubmedQueryRun",
"action_input": "fever, chills, skin rashes causes"
}
```
Observation: Published: 2024-10-02
Title: Rapid identification of primary atopic disorders (PAD) by a clinical landmark-guided, upfront use of genomic sequencing.
Copyright Information: © Dustri-Verlag Dr. K. Feistle.
Summary::
Primary atopic disorders (PAD) are monogenic disorders caused by pathogenic gene variants encoding proteins that are key for the maintenance of a healthy skin barrier and a well-functioning immune system. Physicians face the challenge to find single, extremely rare PAD patients/families among the millions of individuals with common allergic diseases. We describe case scenarios with signature PAD. We review the literature and deduct specific clinical red flags for PAD detection. They include a positive family history and/or signs of pathological susceptibility to infections, immunodysregulation, or syndromic disease. Results of conventional laboratory and most immunological lab studies are not sufficient to make a definitive diagnosis of PAD. In the past, multistep narrowing of differential diagnoses by various immunological and other laboratory tests led to testing of single genes or gene panel analyses, which was a time-consuming and often unsuccessful approach. The implementation of whole-genomic analyses in the routine diagnostics has led to a paradigm shift. Upfront genome-wide analysis by whole genome sequencing (WGS) will shorten the time to diagnosis, save patients from unnecessary investigations, and reduce morbidity and mortality. We propose a rational, clinical landmark-based approach for deciding which cases pass the filter for carrying out early WGS. WGS result interpretation requires a great deal of caution regarding the causal relationship of variants in PAD phenotypes and absence of proof by adequate functional tests. In case of negative WGS results, a re-iteration attitude with re-analyses of the data (using the latest data base annotation)) may eventually lead to PAD diagnosis. PAD, like many other rare genetic diseases, will only be succ
Thought:To determine the possible cause of your symptoms, I will consider common conditions that present with fever, chills, and skin rashes. These symptoms can be associated with a variety of medical conditions, including infections, allergic reactions, autoimmune disorders, and more. Let's look into specific conditions that might be responsible.
Action:
```
{
"action": "PubmedQueryRun",
"action_input": "fever, chills, skin rashes common causes"
}
```
Observation: Published: 2024-10-02
Title: Rapid identification of primary atopic disorders (PAD) by a clinical landmark-guided, upfront use of genomic sequencing.
Copyright Information: © Dustri-Verlag Dr. K. Feistle.
Summary::
Primary atopic disorders (PAD) are monogenic disorders caused by pathogenic gene variants encoding proteins that are key for the maintenance of a healthy skin barrier and a well-functioning immune system. Physicians face the challenge to find single, extremely rare PAD patients/families among the millions of individuals with common allergic diseases. We describe case scenarios with signature PAD. We review the literature and deduct specific clinical red flags for PAD detection. They include a positive family history and/or signs of pathological susceptibility to infections, immunodysregulation, or syndromic disease. Results of conventional laboratory and most immunological lab studies are not sufficient to make a definitive diagnosis of PAD. In the past, multistep narrowing of differential diagnoses by various immunological and other laboratory tests led to testing of single genes or gene panel analyses, which was a time-consuming and often unsuccessful approach. The implementation of whole-genomic analyses in the routine diagnostics has led to a paradigm shift. Upfront genome-wide analysis by whole genome sequencing (WGS) will shorten the time to diagnosis, save patients from unnecessary investigations, and reduce morbidity and mortality. We propose a rational, clinical landmark-based approach for deciding which cases pass the filter for carrying out early WGS. WGS result interpretation requires a great deal of caution regarding the causal relationship of variants in PAD phenotypes and absence of proof by adequate functional tests. In case of negative WGS results, a re-iteration attitude with re-analyses of the data (using the latest data base annotation)) may eventually lead to PAD diagnosis. PAD, like many other rare genetic diseases, will only be succ
Thought:The symptoms of fever, chills, and skin rashes can be associated with a variety of medical conditions. Some common causes include:
1. **Infections**: Viral infections like measles, rubella, or chickenpox can cause these symptoms. Bacterial infections such as scarlet fever or meningitis might also present similarly.
2. **Allergic Reactions**: Allergies to medications, foods, or environmental factors can cause skin rashes along with systemic symptoms like fever and chills.
3. **Autoimmune Disorders**: Conditions like lupus or vasculitis can cause systemic inflammation resulting in fever, chills, and skin manifestations.
4. **Primary Atopic Disorders (PAD)**: These are rare genetic disorders that affect the skin and immune system, making individuals susceptible to infections and other symptoms.
If these symptoms persist or worsen, it is important to consult a healthcare professional for further evaluation and diagnosis. They may consider these and other conditions based on your complete medical history and additional tests.
Final Answer: Fever, chills, and skin rashes can be caused by infections (viral or bacterial), allergic reactions, autoimmune disorders, or rare genetic conditions like primary atopic disorders. A healthcare professional should evaluate persistent symptoms.
> Finished chain.
"Node 'fallback':"
{'documents': [],
'generation': 'Fever, chills, and skin rashes can be caused by infections '
'(viral or bacterial), allergic reactions, autoimmune '
'disorders, or rare genetic conditions like primary atopic '
'disorders. A healthcare professional should evaluate '
'persistent symptoms.',
'question': 'I have fever, chills and skin rashes'}
'\n---\n'
('Fever, chills, and skin rashes can be caused by infections (viral or '
'bacterial), allergic reactions, autoimmune disorders, or rare genetic '
'conditions like primary atopic disorders. A healthcare professional should '
'evaluate persistent symptoms.')Conclusion
By using the approaches demonstrated one can create as many conditionals in langgraph to ensure the accuracy of the output. Various thresholds for evaluating the results can be set. Confidence scores, and relevance scores(>0.8 in the pre-trained model). One can switch to different models where there are better confidence scores.
