The OpenAI RAG system struggled with multiple documents, showing inconsistent performance with our evaluation framework. However, performance improved markedly when all documents were uploaded as a single document. Despite current limitations, such as a 20-file limit per assistant and challenges in handling multiple documents, there is significant potential for improvement. Enhancing the Assistants API to match GPT quality and reducing restrictions could make it a leading RAG solution.
https://www.tonic.ai/blog/rag-evaluation-series-validating-openai-assistants-rag-performance
I’ve had an equivalent of this in a local test service for a while now and one thing I observed was it was really important to give the calling LLM information about when a given document was going to be useful. The usefulness of multiple documents skyrocketed when they were well named and I supplied a one-line description of the content in the main prompt.