AI Document Chatbot for ProjektMagazin.de
We built an AI-powered document chatbot for ProjektMagazin that turns minutes of manual search into seconds of intelligent conversation, giving users instant access to thousands of project management resources.
About ProjektMagazin
What we built
1. Intelligent Question Classification System
We implemented a smart routing system that automatically analyzes each incoming question to determine the best way to handle it.
- When users ask simple conversational questions like "Why are you here?" or "What can you do?", our system recognizes these as generic queries and routes them directly to a lightweight response generator.
- For document-specific questions, the system activates our full retrieval pipeline.
This intelligent classification helps to save on API costs by preventing unnecessary searches through the document database for questions that don't need it.
The result:
The client now achieves faster response times while keeping operational costs predictable and sustainable, even as usage scales.
2. Advanced document indexing with rich metadata
The Droptica team built a comprehensive indexing system that connects directly to ProjektMagazin's Drupal database through JSON APIs. Unlike simple text extraction, our system captures rich metadata with every document:
- Category and taxonomy assignments for precise filtering.
- Author information and IDs for source attribution.
- Custom properties specific to ProjektMagazin's content structure.
This metadata-rich approach enables the chatbot to provide not just accurate answers, but also proper context and source attribution. Users can see exactly which documents and categories their answers come from, building trust and transparency.
The result:
We designed the indexing architecture to be fully extensible. The client can easily add new content types—whether articles, taxonomy terms, or entirely new entity types—without requiring system-wide changes. Each content type has its dedicated endpoint and indexing pipeline, making expansion straightforward and maintainable.
3. Two-stage document grading for superior accuracy
We implemented an innovative two-stage retrieval process that dramatically improves answer quality:
- Stage 1: Broad Retrieval
Our vector search retrieves approximately 20 candidate document chunks that might contain relevant information.
- Stage 2: LLM-Powered Grading
Each candidate chunk is then evaluated by the LLM for actual relevance to the specific question. The system grades each chunk and selects only the top 12 most relevant pieces of information for response generation.
The result:
This approach ensures that users receive answers based on truly relevant content, not just semantically similar text. The client reported significant improvements in answer accuracy and user satisfaction after implementing document grading.
6. AI document chatbot’s built-in security measures
We implemented robust prompt injection protection to guard against malicious attempts to manipulate the chatbot. Our security measures include:
- Input validation to detect and block injection attempts.
- System prompt isolation to prevent unauthorized instruction changes.
- Response filtering to ensure outputs stay within allowed boundaries.
- Continuous monitoring to identify new attack patterns.
The result:
Testing confirmed that the implemented security measures effectively blocked common prompt injection attempts and ensured chatbot responses remained within intended boundaries.
7. Industry-standard tools for monitoring and enhancement
We built the chatbot using industry-standard tools and frameworks that provide robust monitoring capabilities and flexibility for continuous improvement. This approach gave us:
- Complete observability through LangSmith for tracking all LLM interactions and costs.
- User feedback collection built directly into the interface to gather quality ratings on chatbot responses.
- Flexible architecture allowing easy integration of new features and capabilities.
- Production-grade monitoring to identify issues and optimization opportunities.
- Framework independence enabling us to adapt and evolve the system as AI technology advances.
The result:
This tooling infrastructure ensures the chatbot can be continuously monitored, improved, and adapted to emerging best practices without being locked into any proprietary platform. User feedback on response quality provides valuable insights for ongoing system refinement.
Business impact and operational improvements
The client reports high user satisfaction and has already identified visual UI enhancements as the next phase of improvements—a clear signal that the core functionality meets all their needs.
Technology choices
We selected our technology stack based on production reliability, scalability, and long-term maintainability.
Read more about the technical aspects of this project on our blog
In our articles, we describe how we ensured the chatbot’s knowledge stays up to date, how we selected the right tools and frameworks to build a stable RAG pipeline, and which techniques we used to reduce response times and improve the precision of generated answers.
