AI chatbot for internal knowledge sharing

Background

In a large organization, there is often a lot of knowledge, but it is rarely easy to find. Information can be spread across documents, presentations, technical descriptions, and internal guidelines. For the customer's employees, this means that answers to common questions require time, local knowledge, and sometimes help from people who are already heavily burdened.

The customer wanted to investigate how generative AI could be used in a controlled and secure manner to improve access to internal knowledge. The goal was not to build a general chatbot, but a solution that could provide relevant answers based on verified sources and simultaneously function in an enterprise environment with demands for traceability, operability, and further development.

Assignment

The assignment included the development of a customized chatbot that combines searching the customer's internal knowledge base with answer generation via OpenAI. The solution would be able to handle both new questions and follow-up questions in an ongoing conversation.

The assignment has been particularly enjoyable due to the clear business value and the use of AI. We have used the latest AI models and techniques for AI development, which has been very helpful in developing and integrating with AI services, says Anders, AI developer at Arkido.

The work included API development, integration with AWS services, management of conversation history, support for source-based answers, and tools for monitoring quality and usage. An important part was also to create a technical foundation that could be further managed, tested, and evaluated over time.

Solution

The chatbot was built as an ASP.NET Core Web API in .NET 8. Employees at the customer submit a question via a web interface. The solution then retrieves relevant information from documents indexed in Amazon Kendra. This context is then used to create a grounded answer via the OpenAI Responses API.

To reduce the risk of hallucinations, the model is instructed to only use information from the retrieved context material.

The solution also supports conversation chains. Conversation metadata, chat history, and telemetry are stored separately in DynamoDB, providing clear boundaries between user experience, analysis, and OpenAI's stored answers.

In addition to the question flow itself, the solution includes support for sources, labels, and document selection delimitation. Employees or clients can filter the search using labels, making it possible to direct questions to the correct knowledge area. There is also support for screenshots, where visual clues can be used as an aid in troubleshooting.

Technology and working methods

The foundation is built in C#.NET, .NET 8, and ASP.NET Core with API management via Swagger/OpenAPI AI and retrieval: OpenAI Responses API, Amazon Kendra, prompts for source-grounded answers

Storage: DynamoDB for conversations, chat history, and query events

Documentation and metadata are handled with S3 as the document source for indexed material

The system is operated in Docker, AWS ECS Fargate, ECR, CloudFormation, and GitLab CI/CD Quality assurance is managed with xUnit, Moq, unit tests, smoke tests, and a separate evaluation tool

Follow-up and measurement of results are done with a telemetry dashboard, query events, answer quality, fallback rate, and monitoring of response times.

A special evaluation application was developed for regression tests. It can run batches of questions against an environment, compare answers against expected terms, and follow up on, for example, accuracy, fallback behavior, and response times.

Results and business value

Through the customized chatbot, the customer gained a concrete platform to make internal knowledge more searchable, accessible, and useful in everyday life.

The solution contributed to:

• making internal documentation easier to use through natural questions and answers
• relieving support and expert functions from recurring questions
• providing answers based on approved internal sources
• creating traceability regarding usage, answer quality, and information gaps
• laying a technical foundation for continued development of AI support in the organization