Case Study: Dust scales to 5,000+ data sources and sub-second queries with Qdrant
Key results
The challenge
Dust, an OS for AI-native companies that lets users build AI agents on company knowledge, initially created a separate vector collection per data source. As data sources grew beyond 5,000 the approach became unsustainable: RAM consumption skyrocketed and search slowed as memory-mapped vectors spilled to disk, while the team managed nearly a thousand collections and over a million vector operations per cycle.
The solution
Dust migrated to Qdrant, consolidating its fragmented architecture into shared, multi-tenant collections with sharding and payload filtering, and adopted scalar quantization to shrink vector storage while keeping original vectors on disk for fallback.
The results, in context
Queries in large collections that once took 5 to 10 seconds now return in under a second. Collections that once consumed over 30 GB of RAM were optimized to run at a quarter of that size, and the platform scaled to support 5,000+ data sources.