This fella tested the new 128K context window and had some interesting findings.
* GPT-4’s recall performance started to degrade above 73K tokens
* Low recall performance was correlated when the fact to be recalled was placed between at 7%-50% document depth
* If the fact was at the beginning of the document, it was recalled regardless of context length
Any thoughts on what OpenAI is doing to its context window behind the scenes? Which process or processes they’re using to expand context window, for example.
He also says in the comments that at 64K and lower, retrieval was 100%. That’s pretty impressive.
Their needle in a haystack test isn’t very compelling. Sure no test is flawless but a random out of context fact placed at different points in the context window there is a lot of reasons why the model would fail to retrieve that.