Microsoft’s LongRoPE breaks LLM’s context window limit and scales to 2 million tokens
Large-scale language models (LLMs) have achieved remarkable success across a variety of tasks. However, they often suffer from a limited context window size due to high fine-tuning costs, lack of…