You need to generate / hallucinate likely questions/answers as well and then also index embeddings for the questions or queries. So like “what questions can this document answer” kind of stuff. Also I highly recommend using opensearch or similar so you can build hybrid search and not rely just on embedding vectors.

r/singularity

PlayStation fold at home for the next generation.

It’s probably just my own internal bias but I feel like this last week of chaos in this space has resulted in a sudden surge in cool new ideas percolating in the OSS/localllama/ml spaces.

Thanks for sharing this!

Why should I use this instead of opensearch or Postgres?

Being faster/more efficient today isn’t enough to put me off the plugins for these two well known workhorses which bake in years or decades of industry use as well as in cloud and onsite feature parity.

I don’t want to be too harsh but it looks like you’re rebuilding basic DB features whereas Postgres is just finetuning a plug-in.

Why should I use this instead of opensearch or Postgres?

Being faster/more efficient today isn’t enough to put me off the plugins for these two well known workhorses which bake in years or decades of industry use as well as in cloud and onsite feature parity.

I don’t want to be too harsh but it looks like you’re rebuilding basic DB features whereas Postgres is just finetuning a plug-in.

I think a big obstacle is that it is so big hardly anyone is trying to fine tune it.

Plugins for opensearch and Postgres are useful. Dedicated vectordbs are not IMO.

Why did you bury the RTF results in the appendix? These are way more interesting than relative latency compared to vanilla whisper!

One thing I would question: A100GPU might be ‘typical’ for a lot of cloud offerings like what you see coming from DeepGram and the like, but it’s definitely not typical of what you’d find or have access to for any sort of on-site client. Here CPU is still king.

Have you run any experiments with CPU only inference? What does that look like? Here you can still achieve 0.01x-0.06xRT with CPU only inference and basically the same accuracy with a fine-tuned model utilizing the latest production releases from K2/icefall. This looks like it’s getting closer, but I’d still be inclined to recommend using this distilled model to pre-generate a bunch of pseudo labeled training data for a smaller, dedicated K2/sherpa production system for anything on site.

Very cool. Is this still compatible with faster whisper / whisperx and if not how does the speed up compare? Whisperx is already way more than 6x faster using the same v2 models (and still multi lingual).

If we can stack these improvements - awesome!

Will you release your fine tuning framework so others can find tune on their own data?

Does the improvement itself depend on large scale data or can we achieve positive results with smaller fine tuning sets via your approach here?

You can do this pretty seamlessly with hybrid versions of opensearch, combining traditional search queries with vector based embeddings. Vectorize your item descriptions and then turn your free form user queries into hybrid queries that you feed to open search. It’s possible to feed complex natural language queries like “find me some cheap crocs or sandals listed in the last 12 days” and ask an LLM to transform that into a JSON format valid opensearch query. Gpt4 can do this flawlessly if you also provide a JSON Schema that matches your open search index.

I did something very similar recently and I also wasted an enormous amount of time screwing around with haystack, llama index, langchain and the like. I highly recommend avoiding these things if you are trying to build anything custom or that needs to scale or that needs to go onsite.

These frameworks all provide a few nice basic app ideas, and then wrap a laundry list of dedicated vectordbs that I predict will disappear within 2 years. Beyond that they have huge Java like OOP implementation styles that make it a huge pain to hack around or prototype without providing the scalability you’d typically like to trade for that bloat. Plus since they try to treat all the backends the same and this ends up giving more powerful multimodal data stores like Postgres and opensearch short Schrift by not supporting their more useful and sophisticated hybrid search features.

I finally discarded all of it and just started over with an api backend implemented with fastapi and opensearch and oss embeddings.

I’m also wary of OpenAI - gpt4 is great but for this use case it’s best to treat it as a prototyping mechanism and then fine tune a llama2 variant to do the query wrangling. Otherwise it’s too slow, to unreliable (service wise) and way too expensive. There’s no advantage to using OpenAI embeddings.

My 2c. Sounds like a super fun project (at least if it goes like mine has).