Breaking the Bot: A Unified Strategy for Testing GenAI Chatbots
About Topic
Generative AI chatbots are becoming more sophisticated, but how do we ensure their responses are reliable, secure, and contextually accurate? This session will provide a structured strategy for evaluating chatbot quality—combining functional testing for ambiguous queries and systematic evaluation of generated responses into a single, cohesive approach. We’ll explore the core challenges of chatbot quality assurance, including functional ambiguities that arise from complex, multi-turn conversations and the risks of hallucination—where AI generates misleading or incorrect responses. You’ll gain practical techniques to mitigate hallucinations, strengthen retrieval mechanisms, and improve chatbot reliability. Additionally, we’ll dive into security testing, demonstrating real-world vulnerabilities and how adversarial testing can help prevent data leakage, prompt injection, and exploit risks. By the end of this session, you’ll have a clear, actionable strategy to test and improve AI chatbots—ensuring they perform reliably under real-world conditions while maintaining trust, accuracy, and security.
Key Takeaways
- A Unified Testing Strategy – Learn how to combine functional testing for ambiguous queries with systematic evaluation into a single structured approach.
- Minimizing Hallucinations – Understand why AI chatbots hallucinate and discover practical techniques to reduce errors and ensure factual accuracy.
- Security & Vulnerability Testing – Identify real-world security risks such as prompt injection and data leakage, and learn how to proactively test for weaknesses.
- Building Trust & Reliability – Gain actionable insights to improve chatbot response quality, ensuring accurate, relevant, and contextually aware interactions.
About Speaker
Jaydeep Chakrabarty is currently serving as the Director of AI in Tech at Piramal Capital & Housing Finance Limited, based in Bengaluru, Karnataka. With an extensive 18-year career spanning diverse roles such as Fullstack Developer, Infrastructure Developer, Experience and Visual Designer, Project Manager, Quality Analyst, Security Specialist, and Data Scientist, Jaydeep brings a comprehensive technological perspective to his leadership role.
Before joining Piramal Finance, Jaydeep was the Head of Generative AI Engagements and R&D and Head of Communities at Thoughtworks India. At Thoughtworks, he founded and led the Generative AI group “GPT Wizards,” driving research, development, and client partnerships across India, the Middle East, and other global regions. His strategic initiatives significantly contributed to revenue growth by successfully implementing generative AI solutions.
An avid open-source enthusiast, Jaydeep has contributed to notable projects including Talisman, OpenStack, Report-mine, Taiko, and EEG studies. He regularly shares his insights through prestigious publications such as EFY (Electronics For You), Opensource4You, and The Hindu’s tech column “Text and Context – Toolbox.”
Jaydeep’s expertise lies in bridging theoretical advancements in artificial intelligence with practical applications, emphasizing ethical considerations and real-world impact. His current role at Piramal Finance involves integrating AI into critical organizational functions such as credit underwriting and sales enablement.
Recognized as a thought leader in AI innovation and implementation, his contributions have earned him several prestigious recognitions, including the Asia’s Leadership Award 2025, where he was named a “Top Visionary Making a Wave in 2025.” In addition, he was honored as the Best Tech Influencer at the 2nd IT Confex in Dubai in October 2024 and was featured by Insights Success magazine as one of the most impactful and visionary personalities to watch in 2025. He is also a frequent speaker at industry conferences and events, addressing topics related to generative AI, ethical AI deployment, agile methodologies, and emerging technologies.
Other Speakers at QonfX 2025
San Francisco
Atomic Talk Title: 7 Highly Valuable Approaches for REST API Testing
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has
It is a long established fact that a reader will be distracted by the readable conte
will be distracted by the readable content.
Ajay Balamurugadas
Exploratory Testing
Functional Testing
San Francisco
Atomic Talk Title: 7 Highly Valuable Approaches for REST API Testing
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has
It is a long established fact that a reader will be distracted by the readable conte
will be distracted by the readable content.
Ajay Balamurugadas
Exploratory Testing
Functional Testing
San Francisco
Atomic Talk Title: 7 Highly Valuable Approaches for REST API Testing
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has
It is a long established fact that a reader will be distracted by the readable conte
will be distracted by the readable content.
Ajay Balamurugadas
Exploratory Testing
Functional Testing
San Francisco
Atomic Talk Title: 7 Highly Valuable Approaches for REST API Testing
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has
It is a long established fact that a reader will be distracted by the readable conte
will be distracted by the readable content.
Ajay Balamurugadas
Exploratory Testing
Functional Testing