Long Context - Tag - lilfry's libraryhttps://lilfry09.github.io/en/tags/long-context/Long Context - Tag - lilfry's libraryHugo -- gohugo.ioenlilfry@sjtu.edu.cn (lilfry)lilfry@sjtu.edu.cn (lilfry)All rights reserved.Tue, 07 Apr 2026 21:12:00 +0800Read YaRN: Long context expansion, the core is not to pull the window hard, but not to break the RoPEhttps://lilfry09.github.io/en/ai/yarn-context-window-extension/Tue, 07 Apr 2026 21:12:00 +0800fryhttps://lilfry09.github.io/en/ai/yarn-context-window-extension/It would be a pity if we only regard YaRN as “another trick to pull the context from 4k to 64k/128k”.

What it really encounters is a more essential geometric problem than “window expansion”:

**The model is already accustomed to a position and phase system. Now that you forcefully increase the length, how can you make this system continue to work at a longer range while trying not to lose the positional resolution at close range? **

Once this question is clarified, YaRN’s design will appear very natural.

]]>