So I did some research and after I while in the rabbit hole I think that sliding window attention is not implemented in ExLlama (or v2) yet, and it is not in the AMD ROCm fork of Flash Attention yet either.
I think that means it’s just unsupported right now. Very unfortunate, but I guess I’ll have to wait. Waiting for support is the price I pay for saving 900 euros on a GPU by not buying a 4090, but a 7900 XTX. I’m fine with that.
Very interesting. Well, in hindsight I should’ve noticed but performance did decrease after 8k tokens, but became completely unusable after 10k. I actually am pretty disappointed to still know nothing. No one actually documents what does and doesn’t work and when or how. I can barely find anything about SWA (I know what it is in essence) but no one documents how it works, where and if you can set the window size and whether or not it’s in Ooba’s app.
And then there is the problem that I don’t know if it’s supported on AMD cards like you said. Try to look it up, sliding window attention on Google by itself just gives endless pages of “tutorials” and “guides” that don’t tell anything. And combining it with Rocm just gives random results that don’t lead anywhere useful.
Others can help you with the LLM part of this, but I am just mainly curious whether your plan is worth it. You do know that converting an entire book into a summary is pretty much worthless if you don’t make the summary yourself?